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ABSTRACT 


Leadership  is  crucial  in  any  organization.  Executive  leadership  provides  guidance, 
direction  and  structure.  First-line  supervisors  ensure  that  the  vision,  guidance  and 
direction  of  executive  leaders  become  reality  in  an  organization,  and  they  also  are 
responsible  for  the  morale  and  motivation  of  nonsupervisory  employees.  Historically, 
federal  agencies  have  struggled  with  developing  and  implementing  selection  processes 
that  produce  effective  first-line  supervisors. 

This  research  examines  the  four  most  important  tenets  of  a  selection  process. 
These  four  tenets  include  laws  and  regulations  that  govern  first-line  supervisor  selection 
processes,  the  accepted  competencies  associated  with  effective  first-line  supervisors, 
assessment  tools  used  in  the  selection  process  and  most  effective  measures  in  assessing 
organizational  performance  and  first-line  supervisors.  Using  a  multi-method  approach 
that  includes  survey  data,  interviews  and  policy  review,  this  study  compares  first-line 
supervisor  selection  processes  for  effective  and  less  effective  federal  agencies,  as 
measured  by  the  Federal  Human  Capital  Survey  (FHCS)/Federal  Employee  Viewpoint 
Survey  (FedView).  Finally,  conclusions  and  recommendations  are  offered  for  developing 
standards  and  improving  first-line  supervisor  selection  processes  in  DHS  and  the  rest  of 
the  federal  government. 
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EXECUTIVE  SUMMARY 


The  foundation  of  any  organization  is  its  leadership.  Executive  leadership  is  crucial  in 
organizations,  since  such  leaders  provide  guidance,  direction,  and  structure.  However,  it 
is  the  leaders  found  on  the  front  lines,  the  first-line  supervisors,  who  establish  a  stable 
foundation  for  an  organization.  It  is  these  supervisors  who  are  directly  responsible  for 
day-to-day  leadership,  coaching,  mentoring,  job  perfonnance  and  retention  of 
nonsupervisory  employees  (Human  Resources  Management  Panel,  2003).  First-line 
supervisors  ensure  that  the  vision,  guidance  and  direction  of  executive  leaders  become 
reality  in  an  organization.  Because  of  this,  first-line  supervisors  are  considered  an 
important  corporate  leadership  asset.  To  ensure  that  an  organization  best  uses  this  asset, 
leaders  should  dedicate  resources  to  the  selection  and  development  of  first-line 
supervisors.  Historically,  federal  agencies  have  struggled  with  developing  and 
implementing  first-line  supervisor  selection  programs  that  produce  effective  supervisors 
who  positively  contribute  to  the  perfonnance  of  their  agencies  (Human  Resources 
Management  Panel,  2003). 

Federal  agencies  are  required  to  follow  policies,  regulations  and  laws  when 
selecting  first-line  supervisors.  Title  5  Code  Federal  Regulations,  Parts  300,  332,  335, 
and  412,  specifically  addresses  the  selection  and  development  of  first-line  supervisors  in 
the  federal  government;  however,  these  overarching  regulations  are  very  general  and 
provide  no  specific  guidance  for  the  selection  of  first-line  supervisors.  Ultimately, 
federal  agencies  are  responsible  for  developing  their  own  selection  process  to  ensure  the 
general  guidance  of  Title  5  is  followed.  Additionally,  other  regulations  such  as  5  United 
Stated  Code,  Section  3321(a)(2),  provide  for  “a  period  of  probation. .  .before  initial 
appointment  as  a  supervisor  or  manager  becomes  final.”  This  probationary  period  is 
meant  to  evaluate  new  supervisors  to  determine  if  they  are  meeting  the  requirements  of 
the  supervisor  position;  however,  several  government  reports  have  determined  that  this 
probationary  period  in  not  being  used  as  it  was  intended  and  ultimately  is  ineffective. 
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One  report  states,  “The  probationary  period  does  not  appear  to  weed  out  those 
supervisors  who  are  not  performing  well.”  (Office  of  Merit  Systems  Oversight  and 
Effectiveness,  2001,  p.  18) 

Over  the  past  20  years,  several  Office  of  Personnel  Management  (OPM)  and 
Merit  Service  Protection  Board  (MSPB)  reports  have  been  published  that  scrutinize  the 
selection  process  of  first-line  supervisors  (Office  of  Personnel  Management,  2010;  Office 
of  Merit  Systems  Oversight  and  Effectiveness,  2001;  U.S.  Merit  Systems  Protection 
Board,  1989;  U.S.  Merit  Systems  Protection  Board,  2008;  U.S.  Merit  Systems  Protection 
Board,  2010).  However,  it  does  not  appear  that  action  has  been  taken  on  many  of  the 
recommendations  from  these  reports.  There  is  no  consistent  selection  process  for  first- 
line  supervisors  used  by  federal  agencies;  however,  there  is  a  clear  argument  that  every 
federal  agency  is  different  and  therefore  requires  the  latitude  to  implement  their  own 
selection  process.  Much  like  the  probationary  requirement  for  first-line  supervisors,  if 
the  federal  government  does  not  establish  more  than  passive  guidance  for  the  selection  of 
first-line  supervisors,  then  the  effectiveness  of  first-line  supervisors  in  the  federal 
government  is  not  likely  to  improve.  Ultimately,  the  performance  of  each  agency,  and 
the  federal  government  as  a  whole,  will  suffer.  In  this  post- 9/ 1 1  environment  where 
collaboration,  coordination  and  leadership  at  the  frontlines  is  critical  to  the  success  of 
homeland  security,  can  the  Unites  States  afford  to  have  ineffective  first-line  supervisors? 

This  study  compares  first-line  supervisor  selection  processes  for  effective  and  less 
effective  federal  agencies  as  measured  by  the  Federal  Human  Capital  Survey 
(FHCS)/Federal  Employee  Viewpoint  Survey  (FedView).  This  examination  includes 
analysis  of:  (i)  the  selection  processes  used  and  accepted  in  federal  agencies;  (ii)  first-line 
supervisor  competencies  and  (iii)  the  relationship  between  different  selection  processes 
and  first-line  supervisor  effectiveness  in  federal  agencies,  specifically  those  that  impact 
homeland  security. 

Key  results  from  this  study  reveal  several  interesting  findings  that  can  provide 

insights  to  human  resource  personnel,  selecting  officials,  agency  executives  and  scholars. 

As  expected,  the  results  do  not  answer  the  “one”  thing  that  makes  first-line  supervisors 

effective,  but  provides  patterns  and  best  practices  for  those  agencies  involved  in  this 
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study.  These  patterns  and  best  practices  logically  point  to  recommendations  that  federal 
agencies  can  use  to  establish  first-line  supervisor  selection  processes  that  produce 
effective  first-line  supervisors. 

Findings: 

•  Laws  and  regulations  for  first-line  supervisor  selection  are  very  general 
and  only  set  parameters  for  federal  agencies.  Although  OPM  provides 
tools  for  the  selection  process  of  first-line  supervisors,  they  fall  short  on 
setting  specific  steps  or  methods  to  follow  for  the  selection  process  of 
first-line  supervisors. 

•  Agencies  examined  in  this  study  used  competencies  recommended  by  the 
Merit  Systems  Protection  Board  for  first-line  supervisors. 

•  Agencies  used  elements  of  the  first-line  supervisor  selection  process 
provided  by  OPM,  but  the  assessment  tools  used  by  each  agency  greatly 
varied. 

•  When  measured  by  the  Federal  Human  Capital  Survey  (FHCS)/Federal 
Employee  Viewpoint  Survey  (FedView),  agencies  that  fall  under  DHS 
tended  to  have  less  effective  first-line  supervisors. 

•  The  agencies  that  had  less  effective  first-line  supervisors  relied  heavily  on 
online  administered  assessment  tools  and  self-assessment. 

•  The  agencies  that  had  effective  first-line  supervisors  used  multi-method 
selection  processes  that  contained  assessments  tools  that  required 
applicant  interaction  in  demonstrating  behaviors  consistent  with 
competencies  being  sought. 

Recommendations: 

•  OPM  should  assume  a  more  active  role  in  the  selection  process  of  first- 
line  supervisors.  They  should  require  agencies  to  use  assessment  tools 
that  are  comprehensive  in  evaluating  the  behaviors  associated  with  the 
competencies  for  first-line  supervisors.  Agencies  should  not  just  rely  on 
self-assessments  of  past  experience  and  training. 

•  DHS  should  establish  a  common  first-line  supervisor  assessment  test. 
Consistent  with  Government  Accountability  Office  (GAO) 
recommendations  of  implementing  management  functions  and 
strengthening  performance  measurements,  this  test  would  be  a  step  in 
providing  more  stability  and  consistency  in  these  areas  for  agencies  that 
fall  under  DHS.  Ultimately,  this  test  will  cultivate  a  culture  of 
connectedness  and  establish  standards  within  DHS. 

•  Agencies  should  use  a  multi-method  approach  in  selecting  first-line 
supervisors  that  contain  an  assessment  tool  that  requires  applicant 
interaction  but  avoid  self-assessment. 
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•  Using  the  First-Line  Supervisor  Index  (FLSI)  developed  in  this  study, 
agencies  should  begin  to  use  the  FedView  survey  results  for  first-line 
supervisors  to  track  first-line  supervisor  effectiveness  within  their  agency. 

•  By  combining  and  sharing  common  assessment  tools  and  methods, 
agencies  can  lower  the  financial  burden  associated  with  comprehensive 
and  fair  first-line  supervisor  selection  processes. 
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I.  INTRODUCTION 


The  fastest  and  most  direct  way  to  strengthen  Federal  workforce 
performance  is  to  improve  the  supervision  employees  receive. 

Merit  System  Protection  Board  (2010) 


A.  PROBLEM  STATEMENT 

Leadership  is  crucial  in  any  organization.  Executive  leadership  provides 
guidance,  direction,  and  structure.  Leaders  on  the  front  lines,  the  first-line  supervisors, 
establish  a  stable  foundation  for  an  organization.  It  is  these  supervisors  who  are  directly 
responsible  for  day-to-day  leadership,  coaching,  mentoring,  job  perfonnance  and 
retention  of  nonsupervisory  employees  (Human  Resources  Management  Panel,  2003). 
First-line  supervisors  ensure  that  the  vision,  guidance  and  direction  of  executive  leaders 
become  reality  in  an  organization.  Because  of  this,  first-line  supervisors  are  considered  a 
pivotal  corporate  leadership  asset. 

Today,  organizations  should  dedicate  resources  to  the  selection  and  development 
of  first-line  supervisors.  This  will  ensure  that  organizations  better  use  this  asset; 
however,  in  the  past,  federal  agencies  have  struggled  with  developing  and  implementing 
first-line  supervisor  selection  programs  that  produce  effective  supervisors  who  positively 
contribute  to  the  performance  of  their  agencies  (Human  Resources  Management  Panel, 
2003;  U.S.  Merit  Systems  Protection  Board,  2010). 

Historically,  for  most  federal  agencies,  a  first-line  supervisor  was  not  the  primary 
emphasis  of  the  agency.  As  part  of  post- 9/ 11  environment,  first-line  supervisors  more 
than  ever  need  to  motivate  and  mentor  employees  and  encourage  collaboration  and 
coordination  within  the  workplace  and  with  stakeholders.  Additionally,  federal  agencies 
are  becoming  more  decentralized,  relying  more  heavily  on  the  decisions  being  made  at 
the  lowest  level.  Decentralization  creates  a  more  efficient  security  posture,  but  relies 
more  heavily  on  the  first-line  supervisor.  This  only  can  occur  if  the  selection  process 
produces  supervisors  who  have  more  than  just  technical  expertise.  First-line  supervisors 
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need  to  have  and  be  able  to  demonstrate  the  required  competencies  as  soon  as  they  are 
promoted  to  first-line  supervisor,  leading  in  today’s  dynamic  environment. 

Office  of  Personnel  Management  (OPM),  Merit  System  Protection  Board  (MSPB) 
and  several  private  companies  have  refined,  through  research  and  case  studies,  the 
competencies  that  are  most  important  for  first-line  supervisors  (Human  Resources 
Management  Panel,  2003;  Office  of  Merit  Systems  Oversight  and  Effectiveness,  2001; 
Office  of  Personnel  Management,  2010;  U.S.  Merit  Systems  Protection  Board,  1989; 
U.S.  Merit  Systems  Protection  Board,  2010).  Additionally,  this  research  shows  a  strong 
positive  correlation  between  perceived  effectiveness  of  supervisors  and  actual  success  of 
supervisors  and  their  organization  (Brewer,  2005;  O’Driscoll  &  Whitehouse,  2005; 
Thompson,  2007).  However,  research  has  not  been  conducted  that  relates  first-line 
supervisor  selection  processes  employed  by  various  federal  agencies  to  success  and 
effectiveness  of  an  organization. 

The  federal  government’s  selection  process  for  first-line  supervisors  needs  to 
identify  and  select  first-line  supervisors  who  are  effective  and  add  to  the  overall 
performance  of  their  respective  agencies.  It  does  not  appear  that  all  first-line  supervisor 
selection  processes  used  in  the  federal  government  are  accomplishing  this  goal.  If  the 
first-line  supervisor  selection  process  is  not  addressed,  it  will  create  a  leadership  void  that 
is  directly  connected  to  the  success  and  effectiveness  of  federal  agencies. 

Over  the  past  20  years,  several  OPM  and  MSPB  reports  have  been  published  that 

scrutinize  the  selection  process  of  first-line  supervisors  (Office  of  Personnel 

Management,  2010;  Office  of  Merit  Systems  Oversight  and  Effectiveness,  2001;  U.S. 

Merit  Systems  Protection  Board,  2010;  U.S.  Merit  Systems  Protection  Board,  1989;  U.S. 

Merit  Systems  Protection  Board,  2008).  However,  it  does  not  appear  that  action  has  been 

taken  on  many  of  the  recommendations  from  these  reports.  There  appears  to  be  no 

consistent  selection  process  for  first-line  supervisors  used  by  federal  agencies;  however, 

there  is  a  clear  argument  that  every  federal  agency  is  different  and  therefore  requires  the 

latitude  to  implement  their  own  selection  process.  Much  like  the  probationary 

requirement  for  first-line  supervisors,  if  the  federal  government  does  not  establish  more 

than  passive  guidance  for  the  selection  of  first-line  supervisors,  then  the  effectiveness  of 
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first-line  supervisors  in  the  federal  government  is  not  likely  to  improve.  Ultimately,  the 
performance  of  each  agency,  and  the  federal  government  as  a  whole,  will  suffer.  In  this 
post- 9/ 11  environment  where  collaboration,  coordination  and  leadership  at  the  frontline 
levels  is  critical  to  the  success  of  homeland  security,  the  United  States  cannot  afford  to 
have  ineffective  first-line  supervisors. 

B.  FOCUS  OF  THIS  STUDY  AND  RESEARCH  QUESTIONS 

This  thesis  will  use  the  Federal  Human  Capital  Survey  (FHCS)  and  Federal 
Employee  Viewpoint  Survey  (FedView)  to  identify  agencies  that  have  first- line 
supervisors  that  are  more  or  less  effective  and  then  examine:  1.  the  selection  processes 
used  and  accepted  in  federal  agencies;  2.  first-line  supervisor  competencies;  3.  the 
relationship  between  different  selection  processes  and  4.  first-line  supervisor 
effectiveness  in  federal  agencies,  specifically  those  that  impact  homeland  security. 

1.  Research  Question(s) 

Primary:  How  do  first-line  supervisor  selection  processes  vary  in  effective  versus 
less  effective  federal  agencies  as  measured  by  the  FHCS/FedView? 

Sub-questions: 

1 .  What  laws  or  regulations  govern  the  selection  of  first-line  supervisors  in 
government? 

2.  What  competencies  are  most  often  attributed  to  effective  first-line 
supervisors? 

3.  What  important  elements  are  part  of  a  first-line  supervisor  selection 
process? 

4.  Using  the  FHCS/FedView  data  from  years  2006,  2008,  2010  and  a 
composite  first-line  supervisor  index,  which  agencies  have  the  highest  and 
lowest  first-line  supervisor  effectiveness  rating? 

5.  What  are  the  characteristics  of  the  selection  processes  that  tend  to  be  used 
in  federal  agencies  that  have  the  highest  and  lowest  supervisory  ratings? 

C.  SIGNIFICANCE  OF  THIS  RESEARCH 

This  study  will  add  to  the  existing  literature  on  first-line  supervisors  in  the  federal 
government  by  analyzing  the  relationship  of  the  selection  process  to  its  effectiveness  in 
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federal  service.  As  the  federal  government  continues  to  strive  to  produce  a  high- 
performing  workforce,  this  study  could  be  used  to  improve  first-line  supervisor  selection. 
Lastly,  as  homeland  security  agencies  and  stakeholders  evaluate  best  practices  in 
improving  effectiveness  in  homeland  security  related  issues  and  priorities,  this  study  will 
potentially  provide  best  practices  for  selecting  first-line  supervisors  in  their  respective 
organizations  or  agencies. 

D.  LIMITATIONS  OF  THIS  RESEARCH 

There  are  several  potential  shortfalls  and  limitations  associated  with  this  study. 
First,  relying  on  analysis  from  survey  data  to  evaluate  first-line  supervisor  selection 
processes  may  not  provide  a  comprehensive  picture  of  the  causes  and  effects  of  effective 
or  ineffective  selection  processes.  However,  as  mentioned  above,  recent  studies  have 
resulted  in  a  strong  positive  correlation  between  perceived  effectiveness  of  supervisors 
and  actual  success  of  supervisors  and  their  organization  (Brewer,  2005;  O’Driscoll  & 
Whitehouse,  2005;  Thompson,  2007).  This,  coupled  with  interviews  of  employees  who 
are  involved  in  the  first-line  supervisor  selection  process,  will  provide  valuable  insight 
into  the  first-line  supervisor  selection  processes  in  the  federal  agencies  that  are  effective 
and  potentially  ineffective  as  defined  by  a  First-Line  Supervisor  Index. 

In  conducting  interviews  of  employees  involved  in  the  first-line  supervisor 
selection  process,  another  potential  limitation  is  to  assume  all  agencies  have  only  one 
first-line  supervisor  selection  process  for  all  lines  of  business  within  an  agency.  Because 
selection  processes  may  vary,  it  is  important  to  ensure  interviews  capture  as  many  of  the 
first-line  supervisor  selection  processes  for  a  given  agency  as  possible.  This  will  increase 
the  validity  of  the  interview  results  and  their  correlation  to  the  statistical  data. 

Another  shortfall  of  this  research  could  occur  if  this  study  was  taken 
independently  and  not  associated  to  past  research  and  reports  completed  discussing  first- 
line  supervision  in  the  federal  government.  This  study  is  intended  to  provide  additional 
information  and  possible  recommendations  in  improving  first-line  supervisors  in  the 
federal  government  during  a  time  where  more  reliance  on  first-line  supervisors  is 
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required  to  efficiently  address  the  complexities  of  homeland  security.  Therefore,  this 
study  should  be  taken  as  one  piece  of  a  larger  body  of  knowledge  and  research. 

These  shortfalls  and  limitations  are  important  to  keep  in  mind;  however,  in  order 
to  impact  change  in  an  organization  or  process,  new,  innovative  thinking  is  required. 
This  study  will  provide  new  insight  into  how  the  selection  process  for  first-line 
supervisors  impacts  effectiveness  of  first-line  supervisors  in  federal  agencies  and,  in  turn, 
organization  effectiveness. 

E.  ORGANIZATION  OF  THIS  STUDY 

This  study  is  organized  into  five  chapters.  As  indicated  above,  Chapter  I  explains 
the  purpose  and  limitations  of  this  study  and  how  the  topic  of  effective  first-line 
supervisor  selection  processes  will  be  discussed  and  addressed.  Chapter  II  defines  and 
explains  the  four  most  important  tenets  of  a  selection  process  in  the  federal  government 
as  defined  by  pertinent  literature  on  each  tenet.  These  four  tenets  include  the  laws  and 
regulations  that  govern  first-line  supervisor  selection  processes,  the  accepted 
competencies  associated  with  effective  first-line  supervisors,  assessment  tools  used  in  the 
selection  process  and  most  effective  measures  in  assessing  organizational  performance 
and  first-line  supervisors.  Once  the  foundation  of  this  topic  is  established  in  Chapter  II, 
the  research  method  is  explained  in  Chapter  III.  A  multi-method  approach  is  used  in  this 
study  to  examine  effective  first-line  supervisor  selection  processes  in  the  federal 
government.  Chapter  IV  then  examines  the  results  of  the  composite  supervisor  index  and 
the  finding  from  the  interviews.  In  the  last  chapter,  the  author  offers  a  conclusion  and 
recommendations  for  improving  first-line  supervisor  selection  processes. 
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II.  TENETS  OF  FIRST-LINE  SUPERVISORY  SELECTION 


A.  OVERVIEW 

The  first-line  supervisor  selection  process  in  the  government  has  four  important 
tenets.  Similar  to  the  construction  of  an  Egyptian  pyramid,  these  tenets  build  on  each 
other.  If  one  is  missing  or  unstable,  the  others  will  fail.  These  four  tenets  are  described 
in  this  chapter:  1)  the  laws  and  regulations  that  govern  first-line  supervisor  selection 
processes,  2)  generally  accepted  competencies  associated  with  effective  first-line 
supervisors,  3)  assessment  methods  and  tools  used  in  the  selection  processes  and  4) 
overall  measures  to  assess  the  first-line  supervisor  selection  processes. 

B.  LAWS  AND  REGULATIONS  THAT  GOVERN  THE  FEDERAL 

GOVERNMENT  FIRST-LINE  SUPERVISOR  SELECTION  PROCESS 

When  selecting  first-line  supervisors,  federal  agencies,  through  their  authority  to 
manage  personnel  management  functions  within  their  agency,  are  required  to  follow 
federal  policies,  regulations  and  laws.  This  section  describes  the  federal  regulations  that 
address  supervisor  selection,  recruitment,  merit  promotion  and  supervisory  development 
(Title  5  CFR.,  Parts  300,  332,  335,  412),  nondiscriminatory  assessment  methods  (Title  29 
CFR,  Part  1607)  and  the  delegation  of  authority  for  selection  processes  and  probationary 
requirements  (Title  5  USC  Parts  1 104  and  3321). 

1.  Title  5  Code  of  Federal  Regulations  (5  C.F.R.) 
a.  Part  300 

This  regulation  addresses  basic  employment  practices,  general 
employment  information  and  the  identification  and  selection  of  first-line  supervisors. 
This  regulation  also  defines  employment  practices  as  the  “development  and  use  of 
examinations,  qualification  standards,  tests,  and  other  measurement  instruments”  (5 

C. F.R.  Part  300).  It  goes  on  to  suggest  that  measurement  instruments  used  should: 

1 .  Be  practical  in  character  and  as  far  as  possible  relate  to  matters  that  fairly 
test  the  relative  capacity  and  fitness  of  candidates  for  the  jobs  to  be  filled; 
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2.  Result  in  selection  from  among  the  best  qualified  candidates; 

3.  Be  developed  and  used  without  discrimination  because  of  race,  color, 
religion,  sex,  age,  national  origin,  partisan  political  affiliation  or  other 
nonmerit  grounds;  and 

4.  Ensure  the  candidate  opportunity  for  appeal  or  administrative  review,  as 
appropriate.  (5  C.F.R.  Part  300) 

Clearly,  Part  300  provides  parameters  for  selection  of  supervisors,  but  it 
falls  short  of  implementing  specific  required  practices  which  then  leaves  substantial  grey 
area  for  agencies  to  apply  these  regulations  as  needed  within  their  organization. 
Consequently,  first-line  supervisor  selection  processes  are  often  inconsistent  and  lack 
standardization. 

b.  Part  332  and  335 

These  regulations  provide  guidelines  for  recruiting  supervisors  through 
competitive  examination  and  through  promotion  and  internal  placement  respectively. 
Part  335  describes  specifics  related  to  the  merit  promotion  plan  and  merit  promotion 
requirement  guidelines.  In  Part  335(b),  the  regulation  provides  five  requirements  for 
agencies  to  follow  when  promoting  supervisors.  These  requirements  are: 

1 .  Agencies  must  establish  procedures  for  promoting  employees,  which  are 
based  on  merit  and  are  available  in  writing  to  candidates. 

2.  Areas  of  consideration  must  be  sufficiently  broad  to  ensure  the  availability 
of  high  quality  candidates,  taking  into  account  the  nature  and  level  of  the 
positions  covered. 

3.  To  be  eligible  for  promotion  or  placement,  candidates  must  meet  the 
minimum  qualification  standards  prescribed  by  the  Office  of  Personnel 
Management  (OPM).  Methods  of  evaluation  for  promotion  and  placement, 
and  selection  for  training  which  leads  to  promotion,  must  be  consistent 
with  instructions  in  part  300,  subpart  A,  of  this  chapter. 

4.  Selection  procedures  will  provide  for  management's  right  to  select  or  not 
select  from  among  a  group  of  best  qualified  candidates.  They  will  also 
provide  for  management's  right  to  select  from  other  appropriate  sources, 
such  as  reemployment  priority  lists,  reinstatement,  transfer,  handicapped, 
or  Veteran  Recruitment  Act  eligible  or  those  within  reach  on  an 
appropriate  OPM  certificate. 
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5.  Administration  of  the  promotion  system  will  include  recordkeeping  and 
the  provision  of  necessary  information  to  employees  and  the  public, 
ensuring  that  individuals'  rights  to  privacy  are  protected.  Each  agency 
must  maintain  a  temporary  record  of  each  promotion  sufficient  to  allow 
reconstruction  of  the  promotion  action,  including  documentation  on  how 
candidates  were  rated  and  ranked.  These  records  may  be  destroyed  after  2 
years  or  after  the  program  has  been  formally  evaluated  by  OPM 
(whichever  comes  first)  if  the  time  limit  for  grievance  has  lapsed  before 
the  anniversary  date.  (5  C.F.R.  Part  335(b)) 

Although  in  5  C.F.R.  Part  332  and  335  the  government  appears  to  set 
parameters  for  agencies  to  follow  when  selecting  first-line  supervisors,  in  Part  335(b) 
requirement  (4),  the  government  seems  to  leave  the  ultimate  discretion  for  selection  of 
first-line  supervisors  to  the  selecting  manager.  This  regulation  then  presumes  that  the 
“best  qualified  candidates”  have  been  properly  vetted  prior  to  selection  by  the  manager; 
however,  no  regulation  exists  that  provides  for  proper  vetting  of  the  applicants  to  create  a 
pool  of  best  qualified  candidates.  This  is  a  substantial  policy  gap  in  the  current  selection 
process. 


c.  Part  412 

Supervisory  development  is  addressed  in  5  C.F.R.  Part  412.  This  part 
requires  agencies  to  have  written  policies  that  provide  for  the  initial  and  continuing 
development  of  individuals  in  executive,  managerial,  and  supervisory  positions,  and 
candidates  for  those  positions.  The  regulation  does  not  require  pre-supervisory 
development  and  does  not  require  a  vetting  process  when  establishing  selection  processes 
for  first-line  supervisors;  however,  this  regulation  appears  to  provide  for  “pre-selection” 
training  and  development.  Therefore,  as  a  part  of  the  agencies  selection  process, 
potential  candidates  for  promotion  could  be  vetted  through  development  or  training 
activities. 

2.  Title  29  Code  of  Federal  Regulations  (29  C.F.R.) 
a.  Part  1607 

This  part  provides  “Uniform  Guidelines  on  Employee  Selection 
Procedures”  (29  C.F.R.  Part  1607)  specifically  addressing  the  federal  government’s 
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selection  tests  and  other  selection  procedures.  These  guidelines  focus  primarily  on 
ensuring  that  assessment  methods  are  valid  and  are  not  designed  in  a  manner  that 
discriminates  on  the  grounds  of  race,  color,  religion,  sex  or  national  origin.  Although 
extremely  important  as  a  standalone  guideline,  OPM  has  also  incorporated  these 
guidelines  in  the  recommendations  and  guidance  it  issues  on  assessment  methods  and 
tools  for  federal  employees,  to  include  first-line  supervisors  (U.S.  Office  of  Personnel 
Management,  2010). 

3.  5  United  States  Code  (5  U.S.C.) 

a.  Section  1104  (as  Amended  by  Public  Law  1 04-52  of 1 995) 

This  law  delegates  the  authority  of  personnel  management  functions 
within  the  federal  government  to  the  Director  of  OPM  and  authorizes  the  Director  of 
OPM  to  delegate  this  authority  to  agency  heads  and  human  resource  management 
officials  of  each  agency.  In  summary,  5  U.S.C.  §  1104  authorizes  agencies  to  conduct 
personnel  management  functions,  such  as  first-line  supervisor  selection,  within  their 
agency. 


b.  Section  3321(a)  (b) 

This  section  requires  that  new  supervisors  serve  a  probationary  period.  If 

the  new  supervisor  does  not  have  a  successful  probationary  period,  then  the  probationer  is 

placed  back  into  a  position  at  the  same  grade  level  from  which  he  or  she  came.  This 

probationary  period  is  meant  to  evaluate  new  supervisors  to  detennine  if  they  are  meeting 

the  requirements  of  the  supervisor  position.  In  essence,  the  intent  is  to  prevent  or  limit 

the  Peter  Principle  from  occurring.  Laurence  Peter  stated  that  “[i]n  a  hierarchy,  every 

employee  tends  to  rise  to  his  level  of  incompetence”  (Peter  &  Hull,  1969).  The  proper  use 

of  the  evaluations  for  new  supervisors  during  the  probationary  period  allows  agencies  to 

identify  competence  problems  early  on  and  to  take  corrective  action;  however,  several 

government  reports  have  determined  that  this  probationary  period  is  not  being  used  as  it 

was  intended  and  ultimately  is  ineffective  (Office  of  Merit  Systems  Oversight  and 

Effectiveness,  2001;  U.S.  Merit  System  Protection  Board,  2010).  One  report  states,  “the 

probationary  period  does  not  appear  to  weed  out  those  supervisors  who  are  not 

10 


performing  well”  (Office  of  Merit  Systems  Oversight  and  Effectiveness,  2001,  p.  18).  A 
potential  reason  may  be  the  lack  of  valid  first-line  supervisor  evaluations.  Even  if 
evaluations  do  occur,  employees  often  are  not  held  accountable  if  they  fail.  Given  the 
ineffectiveness  of  the  probationary  period,  the  government  should  reexamine  ways  new 
supervisors  can  be  held  to  the  standards  required  for  supervisory  positions  in  the  federal 
government. 

4.  Conclusion 

These  overarching  regulations,  laws  and  guidelines  are  very  general  and  provide 
agencies  with  specific  nondiscriminatory  practices  that  must  be  used  when  hiring  or 
promoting  federal  employees;  however,  it  seems  specific  guidance  and  standards  for  the 
selection  of  first-line  supervisors  is  not  addressed  in  a  fashion  that  ensures  clear 
understanding  of  what  “best  qualified”  means  in  the  selection  process.  Hence,  the 
selection  can  be  flawed.  Additionally,  other  regulations  such  as  5  U.S.C.,  §  3321(a)(2), 
provide  for  “a  period  of  probation. .  .before  initial  appointment  as  a  supervisor  or  manager 
becomes  final.”  Although  this  probationary  period  is  meant  to  evaluate  new  supervisors 
and  to  determine  if  they  are  meeting  the  requirements  of  the  supervisory  position,  it  does 
not  appear  to  be  used  by  agencies  consistently  or  effectively  (5  U.S.C.,  §  3321(a)(2)). 
Ultimately,  federal  agencies  are  responsible  for  developing  their  own  selection  process 
within  the  general  guidelines  of  Title  5  CFR  and  5  USC  and  the  selection  process  for 
first-line  supervisors  in  each  agency  is  likely  to  be  very  different.  This  variability  in 
selection  procedures  is  also  likely  to  lead  to  wide  ranges  of  supervisory  effectiveness. 

C.  FIRST-LINE  SUPERVISORY  COMPETENCIES  IN  THE  FEDERAL 

GOVERNMENT 

Competencies  are  “a  set  of  behaviors  that  encompasses  skills,  knowledge, 
abilities,  and  personal  attributes  that,  taken  together,  are  critical  to  successful  work 
accomplishment”  (Answers.com,  2011).  There  are  several  opinions  as  to  what 
competencies  are  needed  to  be  an  effective  first-line  supervisor.  The  U.S.  Merit  System 
Protection  Board  (2010)  recommends  10  foundational  competencies  for  first-line 
supervisors.  These  competencies  are: 
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•  Accountability 

•  Customer  service 

•  Decisiveness 

•  Flexibility 

•  Integrity/honesty 

•  Interpersonal  skills 

•  Oral  communication 

•  Problem  solving 

•  Resilience 

•  Written  communication  (U.S.  Merit  System  Protection  Board,  2010) 

Additionally,  the  Merit  Board  (2010)  recommends  that  first-line  supervisors 
should  posses  or  have  the  capability  to  possess  the  following  additional  four 
competencies: 

•  Human  capital  management 

•  Developing  others 

•  Conflict  management 

•  Teambuilding 

The  literature  shows  that  it  would  be  impossible  to  establish  a  comprehensive  list 
of  all  competencies  needed  for  all  first-line  supervisors  because  each  organization  (or 
component  of  an  organization)  will  have  its  own  specific  competencies  that  may  be 
unique  to  itself,  or  at  least  not  applicable,  to  all  organizations  or  components.  Therefore, 
in  addition  to  the  above  common  competencies,  each  organization  or  component  should 
conduct  a  job  task  analysis  to  identify  additional  competencies  for  each  first-line 
supervisor  position  (U.S.  Merit  Systems  Protection  Board,  1989,  p.  2;  U.S.  Merit  System 
Protection  Board,  2010).  Job  task  analysis  is  the  process  of  describing  and  recording  task 
specific  aspects  of  jobs  and  specifying  the  skills  and  other  requirements  necessary  to 
perform  the  job. 
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D.  ASSESSMENT  METHODS  USED  BY  THE  FEDERAL  GOVERNMENT 

The  federal  government  uses  a  variety  of  assessment  tools  for  supervisor  selection 
and  promotion.  This  section  describes  U.S.  Office  of  Personnel  Management’s 
recommendations  for  assessment. 

OPM,  whose  tagline  is  “recruiting,  retaining  and  honoring  a  world-class 
workforce  to  serve  the  American  people,”  (2011)  serves  as  a  human  capital  advisor  to 
federal  agencies.  As  such,  they  provide  federal  agencies  with  human  resource 
management  best  practices  and  recommendations  and  they  are  generally  not  responsible 
for  setting  rules,  regulations  or  laws  for  federal  agencies  to  follow.  The  selection  process 
recommendations  can  be  found  on  OPM’s  Personnel  Assessment  and  Selection  Resource 
Center  (U.S.  Office  of  Personnel  Management,  2010).  Included  at  the  Resource  Center  is 
an  automated  assessment  decision  tool  (ADT)  that  links  best  practice  assessment  tools 
from  the  Department  of  Labor. 

OPM  has  consolidated  their  recommendations  and  best  practices  into  a  handbook 
called  Delegated  Examining  Operations  Handbook:  A  Guide  for  Federal  Agency 
Examining  Offices  (2011).  Chapter  II  of  this  handbook  provides  recommendations  on 
selection  and  assessment  methods  for  first-line  supervisors  including  validity  descriptions 
for  each  method,  but  OPM  clearly  states  that  each  process  will  be  different  depending  on 
the  time  frame,  number  of  applicants  and  desired  outcomes  (U.S.  Office  of  Personnel 
Management,  2007).  OPM  recommends  that  each  agency  examine  these  factors  when 
deciding  on  the  assessment  tools  used  to  select  federal  employees,  including  supervisors. 
Additionally,  agencies  are  encouraged  to  use  the  OPM’s  automated  ADT.  This  tool  asks 
questions  regarding  the  expected  outcomes  of  the  selection  process  to  include  desired 
competencies,  number  of  expected  applicants  and  expected  timeframe  for  completion  of 
the  selection  process.  The  ADT  then  provides  recommendations  and  options  comparing 
time  required  for  each  assessment  tool,  the  expertise  needed  for  each  tool  and  the  validity 
of  each  tool.  Table  1  contains  a  list  of  assessment  tools  and  a  comparison  of  the  validity 
and  cost  of  each  test. 
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Table  1.  Example  of  Assessment  Decision  Tool  (ADT)  Results  (From  OPM,  2011) 


Assessment  Method 

Validity^ 

Face  Validity/ 
Applicant 
Reactions^ 

Subgroup 

Differences-! 

Development 

Costs 

Administration 

Costs 

Accomplishment 

Records 

High 

Moderate 

Low 

Moderate 

Moderate 

Assessment  Centers 

Moderate 

High 

Low 

High 

High 

Biographical  Data 

(Biodata)  Tests 

Moderate 

Moderate 

Moderate 

High 

Low 

Cognitive  Ability  Tests 

High 

Moderate 

High 

Moderate 

Low 

Emotional  Intelligence 

Tests 

Moderate 

Moderate 

Low 

High 

Low 

Integrity/Honesty  T ests 

Moderate 

Moderate 

Low 

High 

Low 

Job  Knowledge  Tests 

High 

High 

High 

High 

Low 

Personality  Tests 

Moderate 

Moderate 

Low 

High 

Low 

Reference  Checking 

Moderate 

Moderate 

Low 

Moderate 

Low 

Situational  Judgment 

Tests 

Moderate 

High 

Moderate 

High 

Low 

Structured  Interviews 

High 

High 

Low 

Moderate 

Moderate 

Training  and  Experience 
(T  &  E)  Evaluations 

Low 

Moderate 

Low 

Low 

Low 

Work  Samples  and 
Simulations 

High 

High 

Low 

High 

High 

The  multiple  assessment  tools  mentioned  above  can  be  summarized  in  six  general 


categories: 

1 .  Past  Perfonnance/Experience  Evaluations 

a.  Accomplishment  Records 

b.  Biographical  Data  Tests 

c.  Reference  Checking 

2.  Training  and  Experience  Evaluations 

a.  Practical  Application  Evaluations 

b.  Situational  Judgment  Tests 

c.  Work  Sample/Simulations 


1  Validity  is  the  extent  to  which  assessment  scores  are  related  to  current  or  future  job  performance. 

2  Face  validity  is  the  applicant’s  perception  of  the  validity  of  the  tool  based  on  visual  inspection. 

3  Subgroup  differences  refers  to  the  level  at  which  the  assessment  tool  is  consistent  regardless  of  race, 
sex  or  origin. 
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3.  Written/Standardized  tests 

a.  Cognitive  Ability  Tests 

b.  Emotional  Intelligence  Tests 

c.  Personality  Tests 

4.  Job  Knowledge  Tests 

5.  Interviews 

6.  Assessment  Centers  (OPM,  2011) 

Some  of  the  assessments  methods  recommended  by  OPM  do  not  apply  or  are  less 
applicable  to  supervisor  selection  since  in  most  cases  first-line  supervisors  in  the  federal 
government  are  already  federal  employees.  For  example,  under  the  Americans  with 
Disabilities  Act,  personality  tests  that  meet  the  definition  of  medical  examinations  cannot 
be  required  after  an  offer  for  employment  has  been  made.  Therefore,  personality  tests 
could  not  be  given  to  potential  first-line  supervisors  unless  they  are  new-hires  in  the 
federal  government. 

Although  OPM  provides  multiple  sources,  resources  and  publications  that 
recommend  best  selection  and  assessment  practices,  individual  agencies  ultimately  have 
the  discretion  to  choose  their  assessment  tools.  This  leaves  open  the  possibility  or 
likelihood  that  an  agency  will  choose  an  assessment  tool  or  selection  method  that  will 
meet  the  minimum  federal  regulations  and  guidelines  but  will  fall  short  in  choosing  the 
assessment  tools  and  methods  that  would  have  the  best  chance  of  providing  each  agency 
with  the  best  qualified  candidate.  Perhaps  this  vulnerability  in  the  selection  process  for 
federal  employees,  specifically  first-line  supervisors,  is  an  important  factor  contributing 
to  concern  of  first-line  supervisor  effectiveness  that  has  been  documented  in  several 
OPM,  MSPB  and  independent  organization  studies  (Brewer,  2005;  Human  Resources 
Management  Panel,  2003;  O’Driscoll  &  Whitehouse,  2005;  Office  of  Merit  Systems 
Oversight  and  Effectiveness,  2001). 

E.  MEASURING  FIRST-LINE  SUPERVISORY  SELECTION  PROCESSES 

In  order  for  an  organization  or  a  process  within  an  organization  to  work  properly, 


there  must  be  a  way  of  measuring  its  effectiveness  and  perfonnance.  Without  this  last 

15 


tenet  in  the  selection  of  first-line  supervisors,  agencies  within  the  federal  government 
cannot  detennine  if  the  steps  and  procedures  outlined  in  their  first-line  supervisor 
selection  process  actually  meet  the  desired  individual  and  organization  outcomes. 

An  examination  of  current  literature  for  government  and  private  organizations 
reveals  little  evidence  of  tangible  measures  used  to  gauge  the  effectiveness  of  first-line 
supervisor  selection  processes.  In  private  sector  organizations,  the  selection  processes 
tend  to  be  more  holistic,  viewing  the  selection  and  development  of  first-line  supervisors 
as  one  integrated  process  (Charan,  2008).  Because  of  this,  for  private  companies,  current 
measures  of  effectiveness  become  their  ability  to  fill  first-line  supervisor  positions  with 
qualified  employees  who  have  been  developed  and  assessed  over  a  period  of  time.  The 
ultimate  objective  measurement  of  effectiveness  for  private  organizations,  of  course,  is  its 
ability  to  remain  competitive  and  survive  in  a  free  market  economy.  Unlike  the  private 
sector,  the  federal  government  is  not  concerned  about  profits  and  loss  and  worries  less 
about  survivability.  Therefore,  measuring  effectiveness  in  this  manner  becomes  much 
more  difficult. 

Currently,  it  does  not  appear  that  federal  agencies  measure  the  effectiveness  of 
their  first-line  supervisor  selection  processes  in  any  rigorous  manner.  A  few  studies 
address  perceived  best  practices  or  describe  agencies  going  above  and  beyond  what  is 
required  in  Title  5  Code  of  Federal  Regulations  (CFR)4  (Human  Resources  Management 
Panel,  2003;  Office  of  Merit  Systems  Oversight  and  Effectiveness,  2001),  but  they  do  not 
discuss  a  rigorous  method  of  measuring  the  effectiveness  of  first-line  supervisor  selection 
processes.  Additionally,  OPM  and  MSPB  appear  to  use  extensive  survey  data  in 
assessing  organizational  performance  at  an  agency  level  (U.S.  Merit  System  Protection 
Board,  2010;  U.S.  Office  of  Personnel  Management,  2008),  but  this  data  does  not  appear 
to  be  used  to  measure  more  micro-level  processes  like  first-line  supervisor  selection. 
However,  the  fact  that  MSPB  and  OMP  uses  perceptive  measures  to  evaluate  agency 
performance  and  the  extensive  studies  connecting  perceptive  measures  to  actual 
performance  (Bornmer,  Johnson,  Rich,  Podsakoff,  &  MacKenzie,  1995;  Brewer,  2005; 

4  Title  5  CFR,  Part  300,  332,  335,  412  addresses  the  selection  and  development  of  first  level 
supervisors  in  the  federal  government. 
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Delaney  &  Huselid,  1996;  Dess  &  Robinson,  1984;  O’Driscoll  &  Whitehouse,  2005) 
indicate  that  in  the  absence  of  external  objective  data,  the  best  alternative  way  to  evaluate 
first-line  supervisor  selection  processes  is  by  using  internal  perceptive  measures.  In  order 
for  the  selection  process  of  first-line  supervisors  to  be  most  effective,  however,  the 
federal  government  must  develop  a  measurement  tool  to  evaluate  their  selection 
processes. 

F.  CONCLUSION 

Examination  of  the  four  tenets  of  first-line  supervisor  selection  processes  makes  it 
clear  that  in  order  to  develop  and  maintain  an  effective  selection  process,  several  factors 
need  to  be  taken  into  account  and  constantly  reevaluated  to  ensure  the  process  is  meeting 
the  desired  outcomes.  The  federal  government,  through  laws  and  regulations  and  OPM’ s 
guidance,  has  established  a  framework  that  helps  agencies  address  three  of  the  four  tenets 
of  first-line  supervisor  selection  processes;  however,  the  fourth  tenet  seems  to  be  missing, 
possibly  due  to  the  complexity  and  uncertainty  of  effective  measurement  methods  or 
because  limited  micro  level  external  and  internal  performance  data  exists.  The  fourth 
tenet — measuring  performance  and  organizational  effectiveness  at  an  actionable  level — 
needs  attention  and  substance.  There  is  no  doubt  that  this  area  of  study  is  very  complex 
and  difficult  to  define.  However,  this  difficulty  should  not  preclude  or  in  any  way 
dissuade  scholarly  debate  on  best  practices  and  application  of  the  most  effective  tools  to 
assess  overall  organizational  performance  and  that  of  the  first-line  supervisor  selection 
processes.  Without  all  four  tenets,  first-line  supervisor  selection  in  the  federal 
government  is  likely  to  be  less  effective. 
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III.  RESEARCH  METHOD 


A.  OVERVIEW 

Little  research  has  been  conducted  about  first-line  supervisor  selection  processes 
in  the  federal  government.  Existing  studies  tend  to  report  aggregate  data,  with  analysis 
remaining  at  a  government-wide,  or  agency  level.  Existing  studies  have  not 
demonstrated  how  the  selection  process  of  first-line  supervisors  correlates  with  overall 
agency  effectiveness  or  first-line  supervisor  effectiveness.  Additionally,  existing  studies 
do  not  appear  to  offer  insights  or  solutions  for  improving  the  selection  of  first-line 
supervisors  in  the  federal  government. 

This  study  is  a  multi-method  examination  of  the  selection  processes  for  first-line 
supervisors  in  effective  and  less  effective  federal  government  agencies.  Survey  data  used 
in  this  study  comes  from  the  Federal  Human  Capital  Survey  (FHCS)  and  Federal 
Employee  Viewpoint  Survey  (FedView).  Interview  data  used  in  this  study  comes  from 
in-depth  interviews  and  policy  reviews  about  selection  processes  in  selected  agencies. 
Research  questions  addressed  by  this  study  are  listed  below. 

1.  Research  Questions 

a.  Primary  Research  Question 

How  do  first-line  supervisor  selection  processes  vary  in  agencies  with 
effective  verses  less  effective  first-line  supervisors?  For  each  of  the  sub-questions  below, 
differences  between  agencies  with  effective  first-line  supervisors  and  those  with  less 
effective  first-line  supervisors  will  be  examined. 

b.  Sub-Questions 

1.  What  laws  or  regulations  support  or  hinder  the  selection  of  first-line 
supervisors? 

2.  What  competencies  are  most  often  attributed  to  effective  first-line 
supervisors? 

3.  What  important  elements  are  part  of  the  first-line  supervisor  selection 
process? 
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4. 


Using  the  FHCS/FedView  data  from  years  2006,  2008,  2010  and  a 
composite  first-line  supervisor  index,  how  effective  are  the  first-line 
supervisors  each  agency  and  which  agencies  have  the  highest  and  lowest 
first-line  supervisor  effectiveness  rating? 

5.  What  attributes  of  first-line  supervisor  selection  processes  are  similar  or 
different  in  agencies  with  effective  or  less  effective  first-line  supervisors. 

Analysis  of  data  expected  to  answer  each  of  these  sub-questions  provides 
a  solid  foundation  in  which  to  answer  the  main  research  question  for  this  thesis.  In 
essence,  to  fully  understand  a  problem  and  be  able  to  provide  potential  solutions,  one 
must  first  dissect  and  analyze  the  parts  of  the  problem.  In  doing  this,  the  researcher  will 
be  able  to  suggest  best  practices  for  first-line  supervisor  selection  processes  in  the  federal 
government.  Office  of  Personnel  Management  (OPM)  and  Merit  Systems  Protection 
Board  (MSPB)  research  and  reports  express  concern  within  the  government  about  the 
effectiveness  of  first-line  supervisors  (Human  Resources  Management  Panel,  2003; 
Office  of  Merit  Systems  Oversight  and  Effectiveness,  2001;  U.S.  Merit  Systems 
Protection  Board,  1989).  This  research  will  provide  additional  information  on  one  key 
aspect  of  first-line  supervisor  effectiveness,  the  selection  process. 

It  is  important  to  note  that  this  research  is  not  intended  to  recommend  a 
best  way  to  select  first-line  supervisors  in  the  federal  government.  Several  competing 
factors  influence  and  alter  the  effective  selection  of  first-line  supervisors  beyond  what  is 
discussed  in  this  study;  however,  it  is  expected  that  this  research  will  reveal  insights  into 
how  the  selection  process  impacts  success  of  first-line  supervisors  in  the  federal 
government. 

B.  SURVEY  DATA:  THE  FEDERAL  HUMAN  CAPITAL  SURVEY  AND 

FEDERAL  VIEW  POINT  SURVEY 

In  2002,  OPM  initiated  and  instituted  the  first  Federal  Human  Capital  Survey 
(FHCS).  Since  2002,  OPM  administers  this  survey  every  two  years  to  all  federal 
employees.  In  2010,  OPM  changed  the  name  of  this  survey  to  Federal  Employee 
Viewpoint  Survey  (FedView).  The  FedView  2010  is  an  89-item  survey  that  includes  1 1 
demographic  questions  and  78  items  measuring  federal  employees’  perceptions  about 
how  effectively  the  workforce  is  managed.  Comparing  the  2010  to  the  2008  and  2006 
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FHCS,  several  differences  were  noted.  The  2008  FHCS  had  58  items  in  common  with 
the  2010  FedView  and  the  2006  FHCS  had  57  items  in  common  with  the  2010  FedView. 
When  comparing  the  results  for  each  year,  it  is  important  to  note  that  some  questions  in 
the  overall  survey  and  in  the  demographic  questions  have  changed  or  were  modified 
slightly  in  2010  so  the  data  for  some  questions  needs  to  be  recoded  to  allow  comparison. 
For  example,  one  demographic  question  asked  all  three  years  is:  what  is  your 
supervisory  status  (U.S.  Office  of  Personnel  Managment,  2006)?  The  response  options 
offered  in  2010  were  A.  Nonsupervisor/Team  leader  B.  Supervisor  and  C. 
Manager/Executive.  By  comparison,  in  2008  and  2006,  the  response  options  offered 
were  A.  Nonsupervisor,  B.  Team  Leader,  C.  Supervisor,  D.  Manager,  and  E.  Executive. 
To  be  compared,  responses  for  2008  and  2006  must  be  recoded  (A  and  B  collapsed  to  A, 
and  D  and  E  collapsed  to  C)  to  match  options  offered  for  2010. 

Unlike  with  the  FHCS,  the  2010  FedView  survey  grouped  the  89  questions  into 
eight  topic  areas  that  respondents  see  as  they  take  the  survey.  These  eight  topic  areas  are: 
personal  work  experience,  work  unit,  agency,  supervisor/team  leader,  leadership, 
satisfaction,  work/life,  and  demographics.  In  essence,  by  creating  eight  topic  areas,  OPM 
has  created  indexes  that  can  be  used  to  analyze  the  respondent’s  responses.  For  example, 
the  supervisor/team  leader  topic  area  contained  1 1  questions  that  specifically  pertain  to 
the  perceptions  of  supervisors  and  team  leaders  in  each  agency.5  These  eight  topic  areas 
or  indexes  are  different  and  should  not  be  confused  with  Human  Capital  Assessment  and 
Accountability  Framework  (HCAAF)  indices. 

The  HCAAF  indices  provide  consistent  metrics  for  measuring  progress  toward 
HCAAF  objectives.  The  four  indices  provided  are:  the  Leadership  and  Knowledge 
Management  Index,  the  Results-Oriented  Performance  Culture  Index,  the  Talent 
Management  Index  and  the  Job  Satisfaction  Index.  These  indices  are  used  to  assess 
government-wide  performance  and  trends;  however,  due  to  the  more  general  categories, 
the  HCAAF  indices  are  hard  to  use  for  subsets  or  specific  aspects  of  one  HCAAF  index. 


5  It  is  important  to  note  that  OPM  defines  team  leader  and  supervisor  as  those  who  supervise  or 
oversee  nonsupervisor  employees. 
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Therefore,  in  order  to  examine  first-line  supervisor  performance  in  a  particular  agency,  it 
is  more  pertinent  to  use  the  supervisor/team  leader  topic  area  since  these  questions  are 
more  focused  on  the  effectiveness  of  first-line  supervisors. 

The  2006  and  2008  FHCS  and  2010  FedView  were  self-administered  Web 
surveys  with  paper  version  being  sent  out  to  agencies  that  did  not  have  Web  access.  The 
samples  for  the  FHCS  and  FedView  were  consistent,  based  on  the  number  of  respondents 
and  the  return  rate  for  years  2006,  2008,  and  2010.  The  surveys  all  were  directed  to  full¬ 
time  permanent  employees  and  administered  as  a  census  to  participating  agencies.  The 
participating  agencies  made  up  97  percent  of  the  executive  branch  workforce.  For  all 
three  years,  the  response  rate  was  over  50  percent.  Table  2  shows  the  response  rates  per 
year. 


Table  2.  FHCS/FedView  Response  Rates  (After  OPM,  2006,  2008,  2010) 


Year 

Invited 

Respondents 

Response  Rate 

2006 

390,657 

221,479 

57% 

2008 

417,128 

212,223 

51% 

2010 

504,609 

263,475 

52% 

According  to  OPM,  the  purpose  of  FHCS/FedView  is  to  collect  data  on  the 
perceptions  of  federal  employees  regarding  how  effective  agencies  are  managing  their 
workforce;  however,  OPM  does  not  provide  information  that  connects  perceptions  of 
employees  to  actual  agency  effectiveness.  This  could  be  for  several  reasons,  one  of 
which  is  that  it  is  very  hard  to  determine  government  agency  effectiveness  with  a  metric 
that  can  be  applied  to  all  agencies. 

C.  AGENCY  SELECTION 

Eleven  agencies  were  selected  for  this  study.  The  selection  of  these  agencies  was 
based  upon  similarity  of  the  agencies’  primary  or  secondary  missions  and  their 
relationship  to  homeland  security.  The  following  agencies  are  included  in  this  study. 

•  Bureau  of  Alcohol,  Tobacco,  Firearms  and  Explosives  (ATF) 

•  Drug  Enforcement  Administration  (DEA) 

•  Federal  Bureau  of  Investigation  (FBI) 
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•  Federal  Emergency  Management  Agency  (FEMA) 

•  Transportation  Security  Administration  (TSA) 

•  Citizenship  and  Immigration  Services  (CIS) 

•  Coast  Guard  (CG) 

•  Customs  and  Border  Protection  (CBP) 

•  Immigration  and  Customs  Enforcement  (ICE) 

•  U.S.  Marshals  Service  (USMS) 

•  Secret  Service  (SS) 

D.  DEVELOPMENT  OF  SUPERVISORY  INDEX 

1.  First-Line  Supervisor  Index  (FLSI) 

In  2010,  OPM  modified  FHCS  in  several  ways.  The  result  of  these  modifications 
can  be  seen  in  the  new  version  of  this  survey  called  the  FedView.  In  addition  to  adding 
more  questions  to  the  survey,  OPM  also  grouped  the  new  questions  and  the  old  questions 
that  are  directed  at  a  specific  topic.  As  mentioned  above,  one  of  these  groups  is  called 
“My  Supervisor/Team  Leader.”  Eleven  questions  are  captured  in  this  group.  Of  the 
eleven  questions,  eight  of  these  questions  are  either  identical  or  so  similar  as  to  seem  to 
invoke  the  same  response  from  the  respondent  in  the  FHCS  2006  and  2008. 6 

The  First-Line  Supervisor  Index  (FLSI)  used  in  this  study  contains  the  eight 
questions  listed  in  Table  3.  These  questions  were  used  consistently  in  the  OPM  surveys 
administered  in  2006,  2008,  and  2010.  To  narrow  the  study  to  focus  specifically  on  the 
perceptions  of  nonsupervisor  employees  of  their  first-line  supervisors,  respondents  were 
narrowed  to  only  include  nonsupervisor.  The  following  questions  represent  the  FLSI 
discussed  in  this  study. 


6  Normalization  of  Data-During  the  research  and  analysis  of  the  data  for  the  three  years  included  in 
this  study,  it  was  discovered  that  in  several  instances  the  data  collected  was  not  coded  the  same  for  all 
years.  The  researcher  normalized  the  data,  so  that  accurate  and  complete  analysis  could  be  completed. 
Those  questions  that  applied  to  this  study  and  had  to  be  recoded  and  normalized  can  be  found  in  Appendix 
A. 
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Table  3.  First-Line  Supervisor  Index 


First-Line  Supervisor  Index 

Questions 

Factor 

My  supervisor  supports  my  need  to  balance  work  and  other  life  issues. 

B 

My  supervisor/team  leader  provides  me  with  opportunities  to  demonstrate  my  leadership  skills. 

B 

Discussions  with  my  supervisor/team  leader  about  my  performance  are  worthwhile. 

B 

My  supervisor/team  leader  is  committed  to  a  workforce  representative  of  all  segments  of  society. 

B 

My  supervisor/team  leader  provides  me  with  constructive  suggestions  to  improve  my  job  performance. 

B 

Supervisors/team  leaders  in  my  work  unit  support  employee  development. 

B 

I  have  trust  and  confidence  in  my  supervisor. 

0 

Overall,  how  good  a  job  do  you  feel  is  being  done  by  your  immediate  supervisor/team  leader? 

0 

B=Behavior  Factor  0=Outcome  Factor 

2.  Behavior  Factor  and  Outcome  Factor 

In  developing  this  study,  and  to  understand  the  possible  relationship  between 
questions  within  the  FLSI,  the  researcher  determined  that  within  the  eight  questions  of 
the  FLSI,  two  subgroups  could  be  identified.  These  subgroups  are  the  behavior  factor 
and  outcome  factor.  Although  one  would  assume  that  the  behaviors  and  outcomes  of 
supervisors  would  be  closely  associated,  this  assumption  needed  to  be  tested  to  detennine 
if  these  two  factors  were  indeed  statistically  correlated.  If  perceived  behaviors  of 
supervisors  have  a  strong  correlation  to  the  perceived  outcomes  within  an  agency,  then 
the  overall  FLSI  has  more  statistical  relevance  to  overall  perceived  effectiveness  of  first- 
line  supervisors  in  each  federal  agency  examined.  Additionally,  this  in-depth  validation 
of  the  FLSI  was  important  since  this  index  was  used  in  detennining  how  to  conduct  the 
second  part  of  the  method  in  this  study.  After  examining  the  FLSI  questions,  the 
researcher  found  that  six  of  the  eight  questions  seemed  to  be  asking  about  the  perceived 
behaviors  of  supervisors  or  team  leaders.  These  questions  make  up  the  behavior  factor. 
Two  of  the  eight  questions  asked  about  the  perceived  outcomes  of  supervisors  or  team 
leaders.  These  questions  make  up  the  outcome  factor  (see  Table  3). 

3.  Analysis 

The  FLSI  was  applied  to  all  federal  agencies  included  in  this  study.  The  analysis 
focuses  on  the  nonsupervisor  responses  to  these  index  questions.  Using  analysis  of 


24 


variance  (ANOVA)  and  Pearson’s  correlation  statistical  analysis,  analysis  can  be  focused 
on  the  perceived  effectiveness  of  first-line  supervisors.  ANOVA  and  Pearson’s 
correlation  statistical  tools  are  used  here  because  past  studies  “have  shown  that  measure 
of  perceived  organizational  perfonnance  correlate  positively  with  moderate  to  strong 
associations  with  objective  measures  of  organizational  perfonnance”  (Brewer,  2005,  p. 
511).  This  information  is  presumed  to  identify  which  agencies  have  the  most  effective 
and  the  least  effective  first-line  supervisors.  Using  an  aggregate  mean  of  all  eight 
questions  in  the  FLSI,  the  agencies  were  ranked  from  most  effective  to  least  effective. 

E.  INTERVIEWS 

1.  Data  Collection 

Based  on  the  FLSI  analysis,  interviews  or  policy  reviews  were  conducted  with 
three  agencies  from  the  group  with  the  high  perfonning  first-line  supervisors  and  three 
agencies  were  selected  from  the  group  with  the  lowest  perfonning  first-line  supervisors. 
The  purpose  of  the  interviews  and  policy  reviews  was  to  gather  data  about  the  first-line 
supervisor  selection  process  for  the  selected  agencies.  If  policies  or  procedures  were  not 
available,  in-depth  interviews  were  conducted  with  agency  employees  who  are  familiar 
with  the  agency’s  first-line  supervisor  selection  process.  Generally,  these  employees 
were  human  resource  specialists  directly  involved  in  the  first-line  supervisor  selection 
process  in  each  agency.  A  matrix  of  assessment  method  characteristics  was  used  during 
the  interview  process  or  policy  review  to  capture  the  types  of  assessment  tools  and 
assessment  factors  used  by  each  agency  in  the  sample  (see  Table  4).  Additionally, 
verbatim  notes  were  taken  on  information  provided  in  the  interviews  to  capture  all 
relevant  information  about  the  first-line  supervisor  selection  process. 

2.  Interview  Questions 

When  a  policy  review  was  not  possible  for  an  agency,  the  following  questions 
were  used  during  the  interview  of  an  employee  familiar  with  the  first-line  supervisor 
selection  process  for  the  respective  agency.  The  interviews  were  conducted  on  the 
telephone. 
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1 .  Describe  the  first-line  supervisor  selection  process  in  your  organization. 

2.  Has  the  first-line  supervisor  selection  process  been  changed  or  altered  in 
the  last  five  years?  If  so,  can  you  describe  the  alternations  that  have  taken 
place? 

3.  Is  the  selection  process  you  have  described  consistent  throughout  your 
organization? 

4.  What  aspect(s)  of  the  first-line  supervisor  selection  process  used  by  your 
organization  are  most  important  and  least  important?  Why? 

5.  What  modifications  would  you  make  to  the  current  first-line  supervisor 
selection  process  used  in  your  organization?  Why? 

3.  Selection  Process  Matrix 

Inherent  in  the  recommendations  and  best  practices  provided  by  OPM,  are 
specific  criteria  for  assessing  selection  processes.  The  most  important  criteria  for  this 
study  are  listed  in  the  Table  4  and  used  to  capture  important  characteristics  of  the  first- 
line  supervisor  selection  process  in  each  agency  in  this  study  sample.  These  data  make 
possible  the  comparison  and  analysis  of  first-line  supervisor  selection  processes  and 
identification  of  similarities  or  differences  between  agencies  with  effective  and  those 
with  less  effective  first-line  supervisors. 


Table  4.  First-Line  Supervisor  Assessment  Tool  Matrix 


First-Line  Supervisor  Selection  Assessment  Tools 


Assessment  Tool 

Agency 

1 

Agency 

2 

Agency 

3 

Agency 

4 

Agency 

5 

Agency 

6 

Pre-selection  Vetting 

Number  of  Hurdles  in  Selection 

Process* 

Self-Assessment  Portion  (Y/N) 

Past  Performance  Evaluations 

Accomplishment  Record 

Training  &  Experience 

Practical  Application  Evaluations 

Situational  Judgment  Tests 

Work  Sample/Simulations 

Writing  Skills 

Job  Knowledge  Tests 

26 


First-Line  Supervisor  Selection  Assessment  Tools 

Assessment  Tool 

Agency 

1 

Agency 

2 

Agency 

3 

Agency 

4 

Agency 

5 

Agency 

6 

Interviews 

*See  Appendix  B 

K 

ey 

E=EI< 

Bctronic 

AC=Assessment 

Center 

l=ln-Person 

T=  Telephonic 

4.  Analysis 

The  focus  of  analysis  in  this  study  is  specifically  on  the  elements  and  mechanics 
of  the  first-line  supervisor  selection  process.  The  assessment  tools  and  selection  process 
used  are  examined  to  detennine  similarities  and  differences.  This  includes  examining  the 
numbers  of  hurdles  in  a  selection  process,  if  the  assessment  tools  and  process  are 
administered  electronically,  telephonically  or  in-person,  and  if  there  is  a  self-assessment 
portion  in  the  process.  This  infonnation,  coupled  with  the  FLSI  findings,  is  analyzed  to 
determine  if  there  is  any  relationship  between  the  specific  selection  process  and  first-line 
supervisor  effectiveness  rating. 

F.  CONCLUSION 

Studies  show  that  effective  first-line  supervisors  are  lacking  in  the  federal 
government  which  is  problematic  because  high  perfonning  supervisors  are  critical  to  an 
organization’s  success  and  effectiveness  (Brewer  G.  A.,  2005;  Office  of  Merit  Systems 
Oversight  and  Effectiveness,  2001;  Human  Resources  Management  Panel,  2003;  U.S. 
Merit  Systems  Protection  Board,  1989).  Using  quantitative  and  qualitative  approaches  to 
this  study,  important  insights  about  which  characteristics  of  first-line  supervisor  selection 
processes  are  likely  to  produce  effective  first-line  supervisors.  Contrary  to  existing 
studies  on  first-line  supervisors’  effectiveness  that  focus  at  the  agency  level  and  do  not 
consider  the  nature  of  the  selection  process,  this  multi-method  study  provides  a  starting 
point  for  additional  research  focused  on  this  important  factor — the  selection  process. 
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IV.  RESULTS 


Don’t  rate  potential  over  performance. 


Jim  Fassel 


A.  INTRODUCTION 

The  results  of  this  study  reveal  several  interesting  findings  that  can  provide 
insights  to  human  resource  personnel,  selecting  officials,  agency  executives  and  scholars. 
As  expected,  the  results  do  not  answer  the  “one”  thing  that  makes  first-line  supervisors 
effective  but  provides  patterns  and  best  practices  for  those  agencies  involved  in  this 
study.  The  following  results  emerged  in  response  to  the  research  questions  (proposed  in 
Chapter  III). 

B.  HOW  LAWS  OR  REGULATIONS  GOVERN  THE  SELECTION  OF 

FIRST-LINE  SUPERVISORS  IN  GOVERNMENT 

While  extensive  laws  and  regulations  provide  guidance  and  prohibit  certain 
practices  for  the  selection  of  first-line  supervisors,  these  laws  and  regulations  tend  to  be 
very  general  and  only  set  parameters  for  federal  agencies.  The  Office  of  Personnel 
Management  (OPM)  is  the  federal  agency  responsible  for  providing  specific  guidance 
and  best  practices  based  on  established  laws  and  regulations  for  all  human  resource  issues 
in  the  federal  government.  OPM  provides  tools  for  the  selection  process  of  first-line 
supervisors,  but  it  falls  short  on  setting  specific  steps  or  methods  for  processes  which 
comply  with  the  established  laws  and  regulations.  It  appears  that  it  is  up  to  the  individual 
agencies  to  detennine  what  assessment  methods  and  steps  to  use  when  selecting  first-line 
supervisors  and  that  they  are  doing  so  without  definitive  oversight  regarding  compliance 
with  laws  and  regulations.  This  was  confirmed  during  the  interviews  conducted  in  this 
study.  Each  person  interviewed  understood  that  there  were  laws  and  regulations  that 
governed  the  selection  process,  but  few  actually  knew  the  substance  of  the  laws  and 
regulations.  For  the  most  part,  those  interviewed  referenced  OPM  as  the  organization 
responsible  for  ensuring  agencies  stayed  within  the  parameters  set  by  the  laws  and 
regulations;  however,  little  evidence  from  the  interviews  explained  how  OPM 
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accomplishes  this  regulatory  mission.  This  could  be  due  to  the  fact  that  the  individuals 
interviewed  do  not  have  training  or  experience  with  how  OPM  influences  their  agency. 
While  this  is  interesting  to  note,  agency  compliance  with  laws  and  regulations  is  beyond 
the  scope  of  this  study. 

C.  COMPETENCIES  THAT  ARE  MOST  OFTEN  ATTRIBUTED  TO 

EFFECTIVE  FIRST-LINE  SUPERVISORS 

Competencies  attributed  to  first-line  supervisors  were  explored  through  interviews 
with  ‘savvy  insiders’  and  agency  policy  reviews.  In  general,  ah  agencies  included  in  this 
study  used  at  least  some  of  the  competencies  provided  by  the  Merit  Systems  Protection 
Board  (MSPB),  but  the  number  of  competencies  varied  widely  by  agency  and  specialty. 
Across  ah  six  agencies  and  career  specialties  examined  in  this  study,  the  following 
competencies  were  consistently  sought  for  first-line  supervisors: 

•  Accountability 

•  Flexibility 

•  Integrity/honesty 

•  Interpersonal  skills 

•  Oral  communication 

•  Problem  solving 

•  Written  communication 

It  appears  the  guidance  provided  by  OPM  and  MSPB  with  regards  to  the 
competencies  that  first-line  supervisors  should  possess  is  applied  consistently  across 
agencies  in  this  sample.  Furthennore,  consistent  with  OPM’s  guidance,  it  seems  some 
agencies  cater  to  the  specific  needs  of  their  first-line  supervisor  jobs  incorporating 
additional  competencies  that  go  beyond  the  fundamental  competencies  recommended. 
Even  so,  the  manner  in  which  agencies  assess  these  competencies  for  first-line 
supervisors  seem  to  be  inconsistent. 
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D.  THE  IMPORTANT  ELEMENTS  OF  A  FIRST-LINE  SUPERVISOR 

SELECTION  PROCESS 

All  of  the  agencies  examined  in  this  study  used  elements  of  the  first-line 
supervisor  selection  process  provided  by  OPM  (outlined  in  Chapter  II);  however,  the 
number  of  hurdles  and  the  assessment  tools  used  by  each  agency  and  career  specialty 
greatly  varied.  For  the  most  part,  non-law  enforcement  selection  processes,  compared  to 
law  enforcement  selection  processes,  for  example,  tended  to  use  fewer  assessment  tools 
and  the  process  seemed  to  be  more  streamlined  and  tended  to  be  completed  at  the  field 
office  level.  Additionally,  the  non-law  enforcement  career  specialties  all  had  a  structured 
panel  interview  as  part  of  the  last  hurdle  in  the  selection  process,  whereas  law 
enforcement  career  specialties  did  not  have  this  same  requirement  (see  Appendix  B).  The 
differences  in  selection  processes  for  career  specialties,  specifically  the  number  of 
hurdles  and  assessment  tools  used,  could  be  a  reflection  of  how  each  agency  views  the 
level  of  complexity  for  the  different  career  specialty.  However,  the  data  collected  within 
this  study  does  not  provide  enough  information  to  provide  definitive  infonnation  as  to  the 
reasons  for  the  differences  in  the  selection  processes  for  the  different  career  specialties. 

E.  AGENCIES  WITH  THE  HIGHEST  AND  LOWEST  FIRST-LINE 

SUPERVISOR  EFFECTIVENESS  RATING 

The  First-Line  Supervisor  Index  (FLSI)  discussed  in  Chapter  III  is  a  composite 
measure  of  eight  questions  that  offer  insight  into  the  perceptions  of  nonsupervisor 
employees  about  the  effectiveness  of  their  first-line  supervisors.  Using  the  mean  scores 
of  the  FLSI,  agencies  can  be  ranked  and  an  analysis  of  variance  (ANOVA)7 
accomplished  to  determine  if  these  mean  scores  demonstrate  real  differences  (see 
Appendix  C).  Table  5  shows  the  agencies  FLSI  ranking  per  year  and  is  color  coded  to 
show  top  four  ranks  and  the  bottom  four  ranks. 


7  For  the  purpose  of  this  study,  ANOVA  was  used  to  analyze  differences  in  means  between  agencies. 
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Table  5.  FLSI  Agency  Rank 


FLSI  Rank 

Rank 

2006 

2008 

2010 

1 

ATF 

FBI 

FBI 

2 

DEA 

DEA 

DEA 

3 

FBI 

CG 

ATF 

4 

SS 

ATF 

CG 

5 

CG 

CIS 

SS 

6 

CIS 

SS 

USMS 

7 

USMS 

USMS 

CIS 

8 

ICE 

ICE 

FEMA 

9 

FEMA 

FEMA 

ICE 

10 

CBP 

TSA 

CBP 

11 

TSA 

CBP 

TSA 

As  discussed  earlier,  perceptions  observed  in  survey  data  “have  shown  that 
measures  of  perceived  organizational  performance  correlate  positively,  with  moderate  to 
strong  association,  with  objective  measures  of  organizational  performance”  (Brewer, 
2005,  p.  511).  Although,  for  the  years  studied,  the  difference  between  the  least  effective 
and  most  effective  agencies,  as  defined  by  the  FLSI,  is  no  more  than  four  points,  this  still 
represents  a  significant  statistical  difference  since  in  each  year  there  were  over  9,000 
nonsupervisor  respondents  to  the  survey  for  the  agencies  examined.8  Therefore,  the 
aggregate  mean  is  sufficient  in  ranking  the  agencies  as  most  and  least  effective. 

1.  Correlation  of  Behavior  and  Outcome  Factor 

Using  Pearson  correlation,9  it  was  found  that  the  behavior  factor  (the  aggregate  of 
behaviors  described  in  the  FSLI)  and  outcome  factor  (the  aggregate  of  outcomes 
described  in  the  FLSI)  were  significantly  correlated  above  the  R=0.701  level  for  each 
agency  and  for  each  year  included  in  this  study.  Additionally,  when  combined  into  the 
FLSI,  a  strong  positive  correlation  existed  above  R=0.701  level  for  each  factor  when 

8  To  change  an  agency’s  rank,  several  hundred  respondents  would  need  to  respond  differently. 

9  Pearson  correlation  is  a  statistical  tool  that  tests  linear  dependence  between  two  variables. 
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examined  separately.  This  suggests  that  behaviors  and  outcomes  of  supervisors  are 
closely  connected.  Therefore,  it  follows  that  if  one  assesses  and  selects  supervisors  based 
on  specific  supervisor  behaviors,  then  supervisor  outcomes  will  be  impacted.  The 
correlation  between  behaviors  and  outcomes  is  very  important  and  should  strongly  be 
considered  when  agencies  are  designing  and  implementing  assessment  methods  for  first- 
line  supervisors.  For  details  on  the  Pearson  correlation  results,  see  Appendix  D. 

2.  FLSI  Trends 

Over  the  three  years  examined  during  this  study,  some  interesting  trends  can  be 
observed  for  the  1 1  agencies  included.  First,  FBI,  DEA,  CG  and  ATF  have  consistently 
been  in  the  first  four  spots  for  the  FLSI;  however,  in  2006,  the  SS  ranked  number  four 
and  the  CG  ranked  number  five. 

Second,  there  was  slight  movement  with  regard  to  rank  order  for  the  bottom  four 
agencies,  but  the  same  four  agencies  remained  in  the  bottom  for  all  three  years.  Although 
there  could  be  several  reasons  for  the  agencies  falling  at  the  bottom  of  this  ranking,  to 
include  first-line  supervisor  selection  processes,  it  should  be  noted  that  these  four 
agencies,  ICE,  FEMA,  CBP  and  TSA,  were  all  heavily  impacted  or  reorganized  by  the 
stand-up  of  the  DHS  in  2002. 10 

Figure  1  shows  the  bottom  four  agencies  seem  to  be  improving  as  it  relates  to  the 
FLSI,  with  the  largest  movement  of  over  three  points  by  TSA.  For  TSA,  this  could  be  a 
reflection  of  the  stabilization  of  first-line  supervisor  selection  processes  following  the 
inception  of  the  agency  in  2002.  Although  there  is  generally  a  slight  movement  upward 
for  the  top  four  agencies,  all  four  agencies  had  less  than  one  point  increase  per  year. 
Perhaps  this  is  an  indication  of  consistency  and  effectiveness  of  first-line  supervisor 
selection  processes  in  these  agencies.  Additionally,  none  of  the  1 1  agencies  examined 
exhibited  a  downward  trend  as  defined  by  the  FLSI.  Although  somewhat  unremarkable, 
this  could  demonstrate  that  none  of  the  agencies  examined  have  made  significant  changes 
in  their  first-line  supervisor  selection  processes  that  adversely  impact  the  effectiveness  of 

10  When  these  agencies  were  reorganized,  first-line  supervisor  selection  criteria  may  not  have  been 
clear  and  sometimes  irrelevant  as  more  senior  employees  from  other  agencies  were  integrated  into  these 
agencies — pre-empting  a  rational  selection  process. 
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it  first-line  supervisors.  Lastly,  Figure  1  demonstrates  that  in  all  three  years  included  in 
this  study,  the  top  four  agencies,  as  defined  by  the  FLSI,  scored  above  the  overall  FLSI 
government  mean. 1 1  This  perhaps  provides  credence  to  the  overall  effectiveness  of  first- 
line  supervisors  for  the  top  four  agencies  and  provides  reason  to  look  to  these  agencies 
for  best  practices  in  first-line  supervisor  selection. 
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FBI  DEA  <G  ATF  $$  USMS  CIS  FEMA  ICE  C6F>  T$A 


Figure  1 .  Mean  Rating  of  Agencies  on  Supervisor  Index 


* '  Using  all  nonsupervisor  survey  data  from  each  year  examined,  the  researcher  was  able  to  determine 
the  overall  FLSI  government  mean.  The  overall  mean  in  reflected  in  the  Figure  1. 
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F.  INTERVIEWS/POLICY  REVIEWS 

Based  upon  the  results  of  the  FLSI  (see  Table  5),  “savvy  insiders”  from  the  three 
agencies  in  the  most  effective  group  and  three  agencies  in  the  least  effective  group  were 
interviewed,  or,  in  the  alternative,  first-line  supervisor  selection  policies  were  reviewed. 
The  agencies  examined  in  this  way  were: 

Most  effective: 

•  Bureau  of  Alcohol,  Tobacco,  Firearms  and  Explosives  (ATF) 

•  Drug  Enforcement  Administration  (DEA) 

•  Federal  Bureau  of  Investigation  (FBI) 

Least  effective: 

•  Immigration  and  Customs  Enforcement  (ICE) 

•  Customs  and  Border  Protection  (CBP) 

•  Transportation  Security  Administration  (TSA) 

Although  the  Coast  Guard  (CG)  ranked  in  the  most  effective  group  for  the  FLSI, 
it  was  not  examined  because  it  was  considered  an  outlier  for  purposes  of  this  analysis.  It 
was  treated  as  an  outlier  because  other  than  the  fact  that  CG  falls  within  Department  of 
Homeland  Security  (DHS),  as  an  agency,  it  is  more  like  Department  of  Defense  (DoD) 
agencies  when  selecting  first-line  supervisors.  Like  other  DoD  agencies,  the  CG  has  an 
extensive  vetting  and  selecting  process  for  first-line  supervisors  in  both  the  officer  and 
enlisted  ranks.  The  CG  processes  are  more  holistic  and  akin  to  private  company  selection 
processes  where  development  and  selection  is  a  valued  practice  and  not  a  process  defined 
solely  by  set  times  and  set  assessment  tools.  Although,  like  private  companies,  the 
military  can  provide  significant  insights  to  best  practices  for  selecting  first-line 
supervisors,  its  processes  are  so  different  from  other  agencies  examined  in  this  study,  that 
fair  comparisons  are  not  possible. 
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G.  CHARACTERISTICS  OF  SELECTION  PROCESSES  THAT  TEND  TO  BE 

USED  IN  FEDERAL  AGENCIES  WITH  THE  HIGHEST  AND  LOWEST 

SUPERVISORY  RATINGS 

Using  interviews  and  policy  reviews,  a  selection  process  matrix  was  completed  to 
determine  similarities  and  differences  among  the  agencies  that  were  considered  in  this 
study  to  be  the  most  and  least  effective  as  defined  by  the  FLSI.  Table  6  shows  the  range 
of  assessment  tools  used  by  the  agencies  who  participated  in  this  study.  Although  there 
are  other  assessment  tools  that  are  recommended  by  OPM,  none  of  the  participating 
agencies  use  these  assessment  tools  and,  as  a  result,  they  are  not  included  in  the  matrix. 
Some  of  the  recommended  OPM  assessment  tools  do  not  appear  to  be  used  by  any  of  the 
agencies  examined  include:  standardized  tests  that  evaluate  an  applicant’s  reasoning  and 
logic  and  reference  checks. 

The  effective  and  less  effective  agencies  examined  have  several  similarities  and 
differences  with  each  other  which  can  be  categorized  into  four  areas  1)  use  of  self- 
assessment  tools,  2)  online  administration  of  assessment  tools,  3)  interactive 
demonstration  of  behavior  associated  with  desired  competencies  being  sought  and  4)  use 
of  multiple  assessment  tools.  A  comparison  of  the  six  agencies  follows: 

Table  6.  Results:  First-Line  Supervisor  Selection  Assessment  Tool  Matrix 


First-Line  Supervisor  Selection  Assessment  Tools 


Assessment  Tool 

ATF 

DEA 

FBI 

ICE 

CBP 

TSA 

Non-Law  Enfor 

Special  Agent 

Non-Law  Enfor 

Special  Agent 

Non-Law  Enfor 

Special  Agent 

Special  Agent 

Officer/ Agricul 

Border  Patrol 

Non-Law  Enfor 

Air  Marshal 

Pre-selection  Vetting 

E 

1 

1 

Number  of  Hurdles  in  Selection  Process* 

1 

1 

1 

2 

2 

2 

2 

1 

1 

2 

2 

Self-Assessment  Portion  (Y/N) 

N 

N 

N 

N 

N 

N 

Y 

Y 

Y 

Y 

Y 

Past  Performance  Evaluations 

Accomplishment  Record 

E 

E 

E 

E 

E 

E 

E 

E 

Training  &  Experience 

AC 

AC 

E 

Practical  Application  Evaluations 

Situational  Judgment  Tests 

AC 

AC 

T 

E 

E 

E 
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Work  Sample/Simulations 

AC 

AC 

T 

E 

E 

E 

Writing  Skills 

AC 

AC 

E 

E 

E 

Job  Knowledge  Tests 

AC 

AC 

E 

T 

E 

E 

Interviews 

1 

AC 

1 

AC 

1 

1 

1 

*See  Appendix  B 

Key 

E=Electronic 

AC=Assessment  Center 

l=ln-Person 

T=  Telephonic 

1.  Use  of  Self-Assessment  Tools 

Tools  requiring  self-assessment  were  heavily  used  by  those  agencies  in  the  least 
effective  group.  CBP,  ICE  and  TSA  require  applicants  to  self- assess  their  competencies 
based  on  several  multiple  choice  questions  within  the  assessment  tools  used.  Based  on 
the  available  responses  on  the  self-assessment  questions  reviewed,  an  applicant  can 
determine  what  answer  will  garner  a  higher  score.  These  scores  are  used  as  part  of  the 
overall  first  hurdle  score,  or  in  some  cases,  as  the  only  score  that  is  used  to  determine 
which  applicants  make  it  to  the  next  hurdle.  Therefore,  it  is  possible  that  applicants  will 
inflate  their  experience  when  responding  to  these  self-assessment  questions  so  they  can 
continue  to  be  considered  for  selection  in  the  process.  Moreover,  applicants  who  assess 
themselves  honestly  and  are  potentially  great  first-line  supervisors  may  not  move  to  the 
next  step  of  the  selection  process.  The  agencies  examined  that  fell  in  the  most  effective 
group  did  not  use  any  form  of  multiple  choice  self-assessment  tools.  Instead  these 
agencies  use  assessment  tools  that  allow  for  the  observation  and  assessment  of  patterns  of 
frequent  behaviors  by  applicants.  This  appears  to  be  a  significant  difference  and  might 
be  a  reason  agencies  have  more  or  less  effective  first-line  supervisors. 

2.  Online  Administration  of  Assessment  Tools 

In  every  agency  examined  in  this  study,  it  appears  that  there  is  some  aspect  of  the 
first-line  supervisor  selection  process  that  is  administered  on  a  computer  or  over  the 
Internet.  For  example,  the  initial  submission  of  promotion  packages  appears  to  be 
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universally  completed  over  the  Internet.  The  Internet  has  streamlined  the  application 
process  and  accountability  of  applications;  however,  there  are  some  interesting  findings 
when  agencies  use  computer-based  assessment  tools. 

Two  of  the  three  agencies  that  have  the  least  effective  first-line  supervisors,  ICE 
and  CBP,  use  only  computer-based  assessment  tools  in  their  selection  processes  (see 
Table  6).  Although  these  agencies  use  several  assessment  tools  within  the  selection 
process,  they  do  not  use  any  assessment  tools  that  are  administered  in  a  manner  that  can 
be  observed  by  evaluators.  Their  processes  appear  to  rely  strictly  on  the  responses  an 
applicant  provides  on  the  computer.  Conversely,  other  than  the  initial  submission  of  the 
application,  none  of  the  most  effective  agencies  rely  only  on  computer-based  assessment 
tools.  Typically,  the  effective  agencies  use  a  combination  of  computer-based  assessment 
tools  and  in-person  interactive  assessment  tools.  This  multi-method  approach  is  assumed 
by  the  author  to  provide  a  more  complete  assessment  score  for  the  applicants. 

3.  Interactive  Demonstration  of  Behaviors  Associated  With 
Competencies  Being  Sought 

The  data  suggests  that  in-person  interactive  assessment  tools  are  important  to 
effective  first-line  supervisor  selection  processes.  DEA  and  ATF  have  highly  rated  first- 
line  supervisor  selection  processes  based  on  the  FLSI  ratings  each  year,  and  they  were  the 
only  agencies  examined  that  used  formal  assessment  centers.  Each  agency’s  assessment 
center  administered  slightly  different  assessment  tools,  but  they  both  had  requirements  to 
demonstrate  valued  supervisor  competencies  and  behaviors  in  practical  exercises  and 
simulations.  Although  the  three  agencies  that  were  examined  in  the  least  effective  group 
had  some  practical  exercises,  like  an  in-box  assessment,12  these  assessment  tools  were 
administered  online  and  had  no  interaction  with  role  players  or  evaluators.  The  DEA, 
ATF  and  FBI  each  had  a  portion  of  their  selection  process  that  required  interaction  with 
role  players  and  evaluators  during  the  administration  of  the  selection  process.  The 
interaction  with  role  players  and  evaluators  appears  to  be  a  significant  difference  in  the 


12  An  in-box  assessment  is  an  assessment  tool  used  to  simulate  the  prioritization  of  tasks  and 
assignments  first-line  supervisors  must  conduct  as  part  of  their  supervisory  duties. 
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most  and  least  effective  agencies.  This  finding  provides  important  insight  into  what 
elements  of  selection  processes  appear  to  produce  effective  first-line  supervisors. 

4.  Use  of  Multiple  Assessment  Tools 

One  similarity  among  all  agencies  examined  was  the  use  of  multiple  assessment 
tools;  however,  there  are  differences  in  the  way  the  multiple  assessment  tools  are  used. 
More  effective  agencies  appear  to  use  multiple  assessment  tools  to  confirm  behaviors 
associated  with  required  competencies.  For  example,  ATF  and  DEA  both  use  the 
training  and  experience  evaluation  that  systematically  reveals  and  credits  an  applicant’s 
past  experience.  ATF  and  DEA  then  use  several  other  assessment  tools  like  situational 
judgment  tests  and  simulations  to  confirm  or  discredit  the  finding  from  the  training  and 
experience  evaluation.  Conversely,  TSA,  specifically  the  Air  Marshal  Service,  appears  to 
use  two  assessment  tools  that  reveal  the  same  information  in  a  similar  manner.  The 
Federal  Air  Marshal  Service  (FAMS)  relies  on  accomplishment  records  and  on  training 
and  experience  evaluations.  Both  assessment  tools  rely  on  information  provided  by  the 
applicant  through  essay  style  answers  and  resumes.  So,  unlike  the  mutually  validating 
methods  described  above,  by  using  this  method,  FAMS  appears  unable  to  confirm  that 
the  applicant’s  experience  meets  the  competencies  being  sought.13  In  this  light,  one 
could  argue  for  the  elimination  of  one  of  these  assessment  tools  to  save  overall 
assessment  costs.  Since  the  less  effective  agencies  had  other  factors  mentioned  above, 
like  computer-based  assessments  and  self-assessment  portions  that  appear  to  have 
impacted  their  first-line  supervisor  selection  process  effectiveness,  it  is  not  possible  to 
determine  the  benefit  of  using  multiple  assessment  tools  in  their  selection  process. 
However,  since  effective  agencies  use  multiple  assessment  tools  and  tend  not  to  rely 
solely  on  computer-based  assessment  tools  and  self-assessment  portions  in  the  selection 
process,  one  can  surmise  that  the  use  of  multiple  assessments  tools  that  confirm  applicant 
competencies  in  different  ways  could  enhance  the  effectiveness  of  a  first-line  supervisor 
selection  process. 


13  In  201 1,  FAMS  is  expected  to  revise  its  first-line  supervisor  selection  process,  incorporating 
additional  assessment  tools  that  better  identifies  the  best  qualified  first-line  supervisors  in  this  organization. 
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5. 


Other  Observations 


a.  Agencies  Impacted  by  the  Establishment  of  DHS 

Following  the  development  of  the  Department  of  Homeland  Security  in 
2002,  several  agencies  underwent  significant  reorganization.  CBP,  ICE,  TSA  and  FEMA 
were  all  significantly  impacted  by  the  development  of  DHS.  These  agencies  all  were  in 
the  less  effective  FLSI  group  but  demonstrated  increased  effectiveness  over  the  three 
years  examined  (see  Figure  1).  This  is  most  evident  with  TSA,  which  scored  over  three 
points  higher  from  2006-20 10. 14  Although  none  of  the  agencies  interviewed  in  this 
study  have  significantly  modified  their  first-line  supervisor  selection  process  since  2005, 
it  is  possible  that  the  impact  of  the  stand-up  of  DHS  on  CBP,  ICE,  TSA  and  FEMA  has 
affected  the  effectiveness  of  first-line  supervisor’s  selection  processes  in  each  agency  and 
the  increased  effectiveness  on  the  FLSI  is  an  indication  of  the  stabilization  of  these 
impacted  agencies.  However,  it  can  also  be  argued  that  each  of  these  agencies  have  had 
almost  a  decade  to  improve  their  processes  and  therefore,  at  a  minimum,  should  see 
greater  increase  in  first-line  supervisor  effectiveness  over  the  period  this  study  focuses 
on.  CBP,  ICE,  TSA  and  FEMA  all  fall  well  under  the  government-wide  FLSI  mean 
compared  to  other  agencies  examined  like  SS,  USMS  and  CIS.  Because  of  this,  it 
appears  unlikely  that  the  changes  caused  by  the  standup  of  DHS  were  the  only  reason  for 
these  agencies  having  less  effective  first-line  supervisor  selection  processes. 

b.  Leveraging  Common  Assessment  Tools 

The  implementation  of  each  assessment  tool  has  a  cost  burden  associated 
to  it.  As  depicted  in  Table  1  in  Chapter  II,  OPM  provides  general  insight  into  the 
financial  burden  each  assessment  may  have  on  an  agency.  Interviews  revealed  that  CBP 
and  ICE  appear  to  use  innovative  ways  to  lower  costs  for  each  agency.  It  appears  that 
CBP  and  ICE  leverage  human  resource  personnel  and  common  assessment  tools  to 
counter  act  these  financial  burdens.  For  example,  for  most  first-line  supervisor  selection 
processes  in  CBP  and  ICE,  one  human  resource  office  handles  all  aspects  of  the  selection 

I4  Although  an  increase  of  three  points  does  not  appear  significant,  it  is  important  to  note  that  the 
nonsupervisor  survey  respondents  for  TSA  increased  from  448  in  2006  to  1 156  in  2010.  This  increase  is 
significant  and  makes  TSA’s  three  point  increase  important. 
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process.  This  includes  information  sheets  for  perspective  applicants  and  the  development 
and  selection  of  assessment  tools  used  in  each  process  that  address  competencies  being 
sought.  This  sharing  of  resources  lowers  the  overall  costs  for  each  agency  and  creates 
consistency  among  agencies;  however,  the  sharing  of  resources  appears  to  negatively 
impact  first-line  supervisor  selection  processes  when  assessment  methods  employed  are 
less  effective  since  the  processes  and  assessment  methods  impact  more  than  one  agency. 
In  essence,  perhaps  it  is  the  assessment  tools  used,  not  the  sharing  of  human  resource 
personnel,  which  has  impacted  the  FLSI  rank  for  CBP  and  ICE.  When  effective 
assessment  methods  are  used  by  agencies  that  share  human  resource  personnel,  the  first- 
line  supervisor  effectiveness  will  increase  while  minimizing  the  cost  to  each  agency. 

c.  Use  of  Probationary  Period 

As  indicated  in  the  Chapter  II,  OPM  suggests  that  the  use  of  the 
probationary  period  is  important  and  vital  aspect  of  the  selection  process  for  government 
employees,  specifically  first-line  supervisors.  The  interviews  and  policy  reviews 
revealed  that  none  of  the  agencies  examined  appeared  to  consider  the  probationary  period 
to  be  part  of  the  selection  process.  This  is  consistent  with  past  studies  that  indicated  that 
the  probationary  period  was  not  used  to  weed  out  first-line  supervisors  that  were  not 
meeting  performance  standards  and  goals  for  first-line  supervisors  in  each  agency.  As 
agencies  struggle  to  select  the  most  effective  first-line  supervisors,  it  appears  that  they 
should  look  to  leverage  the  probationary  period  as  a  final  assessment  tool  in  their  first- 
line  supervisor  selection  process. 

H.  CONCLUSIONS 

The  survey  data,  interviews,  and  policy  reviews  conducted  as  part  of  this  study 
provide  significant  insight  into  what  makes  an  effective  first-line  supervisor  selection 
processes  in  the  federal  government.  It  appears  that  the  agencies  examined  follow  the 
laws  and  regulations  when  selecting  first-line  supervisors;  however,  because  OPM  allows 
for  substantial  flexibilities  in  assessment  methods  used  by  each  agency,  the  effectiveness 
of  first-line  supervisor  selection  processes  is  inconsistent.  Additionally,  the  use  of  self- 
assessment  tools,  computer-based  assessment  methods  and  assessment  tools  that  focus  on 
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past  performance,  not  current  behavior,  do  not  tend  to  produce  effective  first-line 
supervisors.  Whereas,  selection  processes  that  use  multi-method  approaches  like  the  use 
of  in-person  interactive  assessment  tools  that  assess  desired  behavior,  appear  to  produce 
more  effective  first-line  supervisors.  Based  on  the  results  of  data  analyzed  in  this  study, 
several  recommendations  can  be  made  that  perhaps  could  improve  the  first-line 
supervisor  effectiveness  in  the  federal  government.  This  will  be  the  subject  of  the  next 
section. 
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V.  CONCLUSIONS  AND  RECOMMENDATIONS 


Not  everything  that  can  be  counted  counts,  and  not  everything  that  counts 

can  be  counted. 


Albert  Einstein 


A.  OVERVIEW 

Bureaucracies  are  needed  because  they  provide  stability,  order  and  protect 
liabilities  (Beetham,  1974).  As  such,  a  bureaucracy  like  the  federal  government  should 
not  change  policies  and  procedures  quickly  or  haphazardly.  By  doing  so,  the  stability  the 
bureaucracy  affords  would  be  jeopardized.  But  in  order  for  the  federal  government  to 
keep  up  with  changes  in  society  and  new  threats,  it  has  to  be  willing  to  incrementally 
adopt  and  update  policies  and  procedures.  This  can  be  done  through  common  sense 
recommendations  that  impact  specific  policies  and  procedures  like  those  affecting  first- 
line  supervisor  selection. 

The  role  and  importance  of  first-line  supervisors  in  the  federal  government  has 
drastically  increased  since  9/11;  however,  many  of  the  policies,  procedures  and  processes 
that  govern  the  selection  of  this  critical  asset  are  outdated  and  ineffective  (Human 
Resources  Management  Panel,  2003;  O’Driscoll  &  Whitehouse,  2005;  Office  of  Merit 
Systems  Oversight  and  Effectiveness,  2001).  These  policies,  procedures  and  processes 
push  federal  agencies  to  avoid  doing  wrong  instead  of  pushing  agencies  to  do  right  in  the 
selection  of  first-line  supervisors.  Although  agencies  and  hiring  officials  need  latitude 
when  selecting  first-line  supervisors,  they  should  also  be  held  accountable  for  the 
methods  and  processes  that  are  used.  Based  on  the  results  of  the  research  conducted  in 
this  study,  several  observations  and  recommendations  are  provided.  These  observations 
and  recommendations  are  intended  to  provide  insights  and  potential  best  practices  to 
federal  agencies,  human  resource  specialists  and  selecting  officials  who  are  interested  in 
improving  the  quality  of  first-line  supervisors  in  the  federal  government. 
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B.  PREDICTORS  OF  EFFECTIVE  FIRST-LINE  SUPERVISORS 

It  has  long  been  argued  that  the  best  predictor  of  future  perfonnance  is  past 
performance.  This  is  why  most  hiring  or  selection  processes  require  evidence  of  past 
experience  and  training,  often  in  the  form  of  a  resume  or  competency  narratives.  Today, 
many  applicants  hire  professional  resume  and  narrative  writers,  and  organizations  with 
limited  resources  use  self-assessments  as  first  hurdles  in  the  selection  process  instead  of 
verifying  an  applicant’s  background  and  experience.  One  might  question  if  this  is  this 
the  best  way  to  select  first-line  supervisors.  Additionally,  these  assessment  tools  usually 
reveal  an  applicant’s  technical  competencies  rather  than  competencies  and  leadership 
potential  that  are  needed  to  be  an  effective  first-line  supervisor.  Unlike  many  private 
companies  that  observe  and  evaluate  employees  for  leadership  potential  as  part  of  a 
development  program  that  starts  when  they  are  first  hired  (Charan,  2008;  Charan,  Drotter, 
&  Noel,  2001),  the  federal  government  tends  to  evaluate  employees  not  on  what  they 
have  the  capability  to  do,  but  rather  what  they  have  done  in  the  past.  This  can  be  a  faulty 
basis  on  which  to  evaluate  employees  because  past  experience  may  have  little  to  do  with 
leadership  potential  or  success  as  a  supervisor.  While  effective  first-line  supervisors 
might  be  selected  by  this  method  through  happenstance,  the  federal  government  is  better 
served  by  purposefully  selecting  the  best  candidates  for  first-line  supervisor  jobs. 

It  is  unlikely  that  federal  agencies  will  be  able  to  accurately  and  fairly  create  an 
environment  similar  to  private  companies  where  leadership  development  is  an 
evolutionary  and  deliberate  process.  Constant  turnover  in  agencies  and  governing  laws 
and  regulations  make  this  difficult,  if  not  impossible,  in  the  federal  government.  So  the 
federal  government  must  build  into  its  first-line  supervisor  selection  process  steps  that 
will  best  position  agencies  for  selecting  the  most  effective  first-line  supervisors.  These 
steps  cannot  be  solely  the  agency’s  responsibility.  They  have  to  be  woven  into  several 
layers  of  the  government  from  the  selecting  official  to  Office  of  Personnel  Management 
(OPM).  This  will  ensure  the  selection  processes  for  first-line  supervisors  are  fair  and 
consistent,  reliably  produce  effective  first-line  supervisors,  and  are  resistant  to  changes  in 
Presidential  Administrations  and  senior  political  appointees  leading  agencies. 
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To  anchor  the  selection  processes  in  all  levels  in  federal  organizations,  the 
government  must  fundamentally  change  the  way  they  view  and  assess  competencies 
required  by  first-line  supervisors.  Clearly,  past  experience  and  training  is  an  important 
aspect  of  any  application;  however,  the  govermnent  cannot  allow  agencies  to  solely  use 
an  applicant’s  reported  background  and  experience.  It  is  merely  a  first  hurdle  in  a 
selection  process.  As  Buckingham  and  Coffman  (1999)  suggest,  past  perfonnance  alone 
is  not  an  indicator  of  future  success.  Predictive  power  comes  from  a  frequent  pattern  of 
the  behavior.  This  predicative  power  only  can  be  uncovered  through  assessment  tools  that 
reveal  the  frequent  and  demonstrated  patterns  of  behavior  that  are  being  sought  as  part  of 
the  designated  competencies.  This  study  suggests  specific  action  that  should  be  taken  by 
any  agency  or  leader  who  wants  to  produce  effective  leadership  at  the  first-line 
supervisor  level. 

C.  THE  ROLE  OF  OPM  IN  FIRST-LINE  SUPERVISOR  SELECTION 

As  the  human  resource  agency  within  the  federal  government,  OPM  should 
assume  a  more  active  role  in  guiding  design  of  the  selection  process  of  first-line 
supervisors.  That  is,  they  should  require  agencies  to  do  more  than  just  rely  on  past 
experience  and  training.  On  May  11,  2010,  President  Obama  issued  a  federal  hiring 
reform  memorandum.  The  guidance  in  the  memorandum  called  for  modifying  the  hiring 
and  recruiting  of  federal  employees.  Key  modifications  were: 

•  Elimination  of  written  essays  (KSAs) 

•  Allow  individuals  to  apply  with  resume  and  cover  letter 

•  Use  category  rating 

•  Ensure  manager  responsibility  and  accountability  for  hiring 

•  Improve  quality  and  speed  of  hiring  (Obama,  2010) 

Although  this  memorandum  is  not  directed  solely  at  first-line  supervisor  selection, 
it  demonstrates  the  Administration’s  concern  and  focus  on  federal  human  capital 
activities.  The  established  reform  guidance  and  subsequent  actions  taken  by  OPM  to 
ensure  this  guidance  is  implemented  represents  significant  steps  in  improving  the  federal 
workforce;  however,  if  one  reviews  the  Fiscal  Year  2010  Agency  Financial  Report 
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published  by  OPM,  it  is  clear  that  although  OPM  is  focused  on  the  federal  hiring  reform, 
the  primary  focus  of  OPM  does  not  include  first-line  supervisor  selection  (United  States 
Office  of  Personnel  Management,  2010).  By  not  putting  more  emphasis  on  first-line 
supervisors,  the  reforms  that  President  Obama  is  requiring  will  fail.  Effective  first-line 
supervisors  are  needed  to  mentor,  lead,  and  provide  guidance  to  the  federal  employees 
President  Obama  is  targeting  with  this  hiring  reform.  Concurrent  to  the  reforms  required 
by  the  President,  OPM  must  examine  and  move  to  refonn  its  role  in  the  selection 
processes  at  the  first-line  supervisor  level,  if  it  wants  to  meet  the  second  part  of  the 
agency’s  strategic  mission  of  retaining  and  honoring  the  federal  workforce.  This  is 
because  studies  have  shown  that  employees  will  stay  in  an  organization  primarily  based 
on  their  supervisor.  Hence,  there  is  more  at  stake  than  selection  and  development  of  first- 
line  supervisors.  Retention  of  the  workforce  is  at  risk  as  well.  Specifically,  in  a  2004 
Corporate  Leadership  Council  study,  50,000  employees  worldwide  indicated  that  22  of 
the  top  25  drivers  of  employee’s  retention  at  their  company  were  supervisor-led 
(Thompson,  2007,  p.  3). 

It  is  unrealistic  for  OPM  to  completely  take  over  or  overhaul  the  first-line 
supervisor  selection  process  in  a  centralized  manner;  however,  by  refining  first-line 
supervisor  selection  guidance,  OPM  can  effect  change  in  first-line  supervisor  selection 
processes  without  taking  control  and  autonomy  away  from  the  agencies  that  OPM  serves. 

While  OPM,  as  discussed  in  Chapter  II,  provides  extensive  assessment  tools  and 
resources  to  federal  agencies,  interviews  and  policy  reviews  conducted  during  this  study 
suggest  OPM  leaves  the  decision  about  use  of  these  tools  to  the  various  agencies. 
Although  this  decentralized  approach  provides  enormous  flexibility  to  agencies  to 
develop  first-line  supervisor  selection  processes  customized  to  an  agency’s  unique  needs, 
it  creates  an  environment  that  does  not  cultivate  effective  selection  processes  needed  or 
selection  consistency. 

Perhaps  OPM  should  implement  a  system  that  provides  more  stringent  regulations 
for  the  assessment  and  selection  methods  used  to  select  first-line  supervisors.  Using  best 
selection  practices,  some  of  which  were  examined  in  this  study,  OPM  could  provide 

minimum  assessment  method  options  based  on  constraints  identified  by  each  agency. 
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Instead  of  just  infonning  agencies  of  the  assessment  tools  pros  and  cons  based  on  agency 
restraints,  OPM  should  require  minimum  assessment  methods  that  are  needed  based  on 
certain  restraints.  For  example,  if  an  agency  has  several  hundred  possible  applicants  for  a 
supervisory  position  that  needs  to  be  fdled  within  60  days,  perhaps  OPM  could  provide  a 
minimum  set  of  assessment  tools  that  agencies  should  chose  from  that  would  accomplish 
their  goal;  yet  at  the  same  time,  ensure  selection  processes  that  have  the  best  opportunity 
of  identifying  the  best  qualified  applicants  and  that  would  prevent  agencies  from  only 
using  self-assessments  and  competency  narratives  in  evaluating  applicants. 

By  creating  minimum  selection  process  criteria,  OPM  would  take  a  substantial 
step  in  ensuring  first-line  supervisor  selection  processes  across  agencies  are  more 
effective  than  the  processes  that  are  currently  used.  This  minimum  selection  criteria 
would  allow  for  agencies  to  continue  crafting  selection  processes  to  fit  their  individual 
needs  but  create  a  level  of  consistency  in  the  federal  government  that  will  strengthen  its 
foundation — the  first-line  supervisors. 

D.  CREATING  CONNECTEDNESS  WITHIN  DHS 

This  study  revealed  agencies  that  have  the  least  effective  first-line  supervisor 
selection  processes  are  those  impacted  by  the  stand  up  of  DHS,  specifically  FEMA,  CBP, 
ICE  and  TSA.  This  is  one  more  piece  of  evidence  that  indicates  many  of  the  agencies  that 
fall  under  DHS  struggle  to  stabilize  and  create  effective  processes.  Clearly,  as  many 
Government  Accountability  Office  (GAO)  reports  have  indicated,  there  are  several 
potential  steps  that  will  create  more  stability  and  consistency  for  the  agencies  that  fall 
under  DHS  (U.S.  Governement  Accountability  Office,  2008;  2010). 

Most  of  the  recommendations  provided  by  GAO  are  at  a  strategic  level  such  as 
implementing  management  functions  and  strengthening  perfonnance  measurements  in 
DHS.  Since  DHS  agencies  tended  to  score  lower  on  perceptive  measures  of  effectiveness 
(the  FLSI),  perhaps  DHS  can  make  strides  in  accomplishing  these  strategic 
recommendations  by  focusing  on  a  specific  process  such  as  the  first-line  supervisor 
selection  process,  one  that  impact  all  agencies  within  DHS.  As  DHS  continues  to 
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establish  itself  as  a  cabinet  level  agency  and  as  the  culture  builds  within  this  agency,  it  is 
critical  to  establish  common  threads  and  standards,  wherever  possible,  fostering  a  unified 
DHS. 

A  common  first-line  supervisor  assessment  process  would  be  an  important  step  in 
creating  a  more  connected  DHS  culture  in  which  ah  the  sub-agencies  use  the  same 
process  to  select  employees  for  leadership  responsibility.  Similar  to  the  Treasury 
Department’s  Treasury  Enforcement  Agent  Exam  (TEA)  and  Department  of  Defense’s 
Armed  Services  Vocational  Aptitude  Battery  (ASVAB),  DHS  should  create  a  common 
assessment  test  that  ah  potential  first-line  supervisors  within  DHS  must  pass.  Granted, 
the  TEA  and  ASVAB  are  administered  for  entry-level  positions,  but  what  is  important 
and  can  be  applied  to  the  first-line  supervisor  selection  process  is  the  consistency  and  a 
common  thread  that  the  ASVAB  and  TEA  creates.  Regardless  of  the  military  branch  or 
military  specialty,  every  applicant  must  take  the  ASVAB.  Each  military  branch 
determines  how  to  use  the  ASVAB  score  to  determine  the  military  specialty  for  which  the 
applicant  is  eligible.  This  common  test  is  required  by  ah  military  branches  and  creates 
consistency  and  a  common  evaluation  standard.  By  employing  a  common  first  hurdle 
assessment  test  for  first-line  supervisors,  DHS  will  create  more  connectedness  within  the 
department. 

As  discussed  in  the  results  section  of  this  study,  the  use  of  assessment  tools  varies 
greatly  among  the  agencies  examined.  Although  the  agencies  need  latitude  to  structure 
the  selection  processes  that  best  meets  the  competencies  and  qualifications  needed,  there 
are  benefits  to  having  a  baseline  hurdle  that  ah  first-line  supervisors  must  go  through  in 
order  to  be  considered  for  specific  positions  in  a  given  agency.  For  example,  there  are 
several  Psychometric  tests  like  cognitive  ability15  and  emotional  intelligence16  tests  that 
can  be  administered  online  and  could  provide  several  benefits  to  DHS,  its  sub-agencies 


15  Cognitive  ability  tests  test  abilities  involved  in  thinking  (reasoning,  perception,  memory,  verbal,  and 
mathematical  ability  and  problem  solving). 

16  Emotional  intelligence  (El)  test  is  defined  as  a  type  of  social  competence  involving  the  ability  to 
monitor  one’s  own  and  other’s  emotions,  to  discriminate  among  them,  and  to  use  the  information  to  guide 
one’s  thinking  and  actions.  El  is  fairly  specific  ability  that  connects  a  person’s  knowledge  processes  to  his 
or  her  emotional  processes. 
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and  selecting  officials.  These  psychometric  tests  evaluate  an  applicant’s  abilities 
involved  in  thinking,  to  include  reasoning,  perception,  memory,  and  problem  solving  and 
how  these  abilities  are  connected  to  his  or  her  emotional  processes.  A  standard 
psychometric  test  would  provide  the  following  benefits  to  the  agencies  that  fall  within 
DHS: 

1.  Provide  valuable  demographic  infonnation  to  OPM,  DHS  and  sub¬ 
agencies  that  could  be  important  in  analyzing  the  federal  workforce. 

2.  Increase  face  validity17  of  first-line  supervisor’s  selection  processes  across 
the  federal  government,  which  impacts  overall  morale  and  support  of  first- 
line  supervisors. 

3.  Create  a  first-line  supervisor  competency  baseline  that  gives  insight  into 
competencies  being  sought  by  each  agency. 

4.  Provide  a  strand  of  unity  among  all  first-line  supervisors.  Therefore,  if  a 
first-line  supervisor  desires  to  changes  agencies,  the  receiving  agency 
could  use  these  scores  as  a  first  step  in  determining  suitability. 

5.  Remove  part  of  the  financial  burden  from  agencies  and  ensures  the 
reevaluation  and  modification  of  such  assessment  tests  could  be  uniformly 
updated  and  applied  across  the  federal  workforce  in  DHS. 

This  first-hurdle  assessment  test  would  not  require  a  mandatory  elimination  of 
potential  first-line  supervisors  based  on  a  set  minimum  score.  Instead,  similar  to  the 
military  and  the  Treasury  Department,  the  agency  the  applicant  is  applying  for  would 
receive  the  complete  list  of  names  with  the  scores  from  the  psychometric  test(s).  It  would 
be  up  to  each  agency  to  determine  how  these  results  would  factor  into  the  first-line 
supervisor  process  they  have  in  place.  For  example,  if  an  agency  uses  self-assessments, 
interviews  or  situational  judgment  tests  in  the  selection  process,  the  psychometric  test 
results  could  be  compared  to  these  results  to  validate  findings  from  these  assessment 
tools.  This  will  provide  insights  into  specific  applicants  that  an  agency  might  want  to 
further  investigate. 

With  over  200,000  employees,  the  DHS  workforce  is  large,  which  makes  it  hard 
to  create  and  ensure  consistency  in  any  human  resource  function.  By  creating  an  online 

17  Face  validity  is  how  a  process  appears  at  “face  value.”  Does  a  process  appear  to  select  first-line 
supervisors  in  a  fair  and  unbiased  way? 


49 


centralized  common  first-hurdle  test  for  first-line  supervisors,  perhaps  this  would  assist 
DHS  in  creating  consistency  in  first-line  supervisor  selection  processes  and  begin  to 
establish  overall  connectedness  within  the  department. 

E.  SELECTION  PROCESS  BEST  PRACTICES 

As  stated  before,  this  study  is  not  intended  to  suggest  a  cookie-cutter  selection 
process  for  selecting  effective  first-line  supervisors.  This  would  discount  several  factors 
that  influence  the  effectiveness  of  first-line  supervisors  that  are  outside  the  scope  of  the 
selection  process.  But  the  results  of  this  study  suggest  there  are  several  best  practices  that 
surface  for  selecting  first-line  supervisors. 

1.  Avoid  Self  Assessment 

Self-assessments  produce  potentially  flawed  results.  If  an  agency  uses  self- 
assessments,  they  should  be  used  as  an  assessment  tool  only  in  the  first  hurdle  of  the 
selection  process.  Most  applicants  know  that  to  be  considered  and  make  it  through  the 
first  hurdle  of  a  selection  process,  a  certain  score  has  to  be  obtained  through  self- 
assessment.  Therefore,  there  is  no  drawback  for  an  applicant  to  inflate  the  rating.  Self- 
assessment  tools  allow  for  dishonest  applicants  to  make  it  to  the  next  hurdle  with  little 
fear  of  being  held  accountable;  whereas  honest,  potentially  great  first-line  supervisors 
may  not  make  the  cut.  This  dramatically  impacts  morale  in  an  organization  since  the 
self-assessments  may  be  prone  to  having  unqualified  applicants  make  it  to  the  next 
hurdle,  whereas  potentially  qualified  applicants  may  get  disqualified  and  are  unable  to 
advance.  This  study  reveals  that  the  agencies  that  were  rated  as  less  effective  relied  on 
self-assessment  as  part  of  their  first  hurdle  for  selection  of  first-line  supervisors.  Doing 
away  with  self-assessments  tools,  agencies  could  save  money  or  invest  in  more  valid 
assessment  tools. 

2.  Applicant  Interaction  Assessment  Tools 

The  agencies  defined  as  effective  in  this  study  all  had  some  applicant  interaction 
assessment  tools  used  in  their  selection  process.  The  FBI’s  interaction  was  through  a 
telephone  administered  in-box  assessment  tool,  whereas  the  DEA  and  ATF  both  used 
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assessment  centers  that  contained  multiple  assessment  tools,  many  requiring  in-person 
observations  of  desired  competencies  and  behaviors.  Observation  assessment  tools  such 
as  interviews,  practical  exercises  and  simulations,  provide  the  connection  between  what 
an  applicant  has  the  potential  to  do  based  upon  his  or  her  training  and  experience,  and 
what  an  applicant  can  currently  demonstrate.  This  allows  evaluators  and  assessors  to 
observe  current  behaviors  that  actually  apply  to  required  and  important  supervisor 
competencies.  Additionally,  more  applicant  interactive  assessment  tools  could  help 
eliminate  the  federal  government’s  habit  of  selecting  first-line  supervisors  based  on 
technical  expertise  instead  of  supervisor  capabilities  and  competencies. 

3.  Multi-Method  Approach 

More  than  one  selection  assessment  tool  is  needed  in  any  first-line  supervisor 
selection  process  in  the  federal  government.  Although  the  results  of  this  study  seem 
somewhat  inconclusive  regarding  the  number  of  hurdles  and  assessment  tools  needed,  it 
is  clear  agencies  with  more  effective  first-line  supervisors  (rated  by  the  FLSI)  used 
multiple  assessment  tools,  did  not  rely  solely  on  self-assessments  and  used  assessment 
tools  that  required  applicant  interaction.  Additionally,  this  multi-method  approach  is 
consistent  with  recommendations  from  OPM  and  the  Department  of  Labor  (United  States 
Department  of  Labor,  2000;  U.S.  Office  of  Personnel  Management,  2007).  In  short,  this 
approach  provides  the  clearest  picture  of  a  candidate’s  abilities  in  terms  of  competencies 
being  sought  because  is  reveals  frequent  patterns  of  behavior.  Most  private  companies  do 
not  have  set  selection  hurdles  or  tools  in  their  first-line  supervisor  selection  processes 
since  they  evaluate  potential  first-line  supervisors  over  a  period  of  time  instead  of  a  set 
selection  process.  However,  since  the  federal  government  uses  a  set  selection  process,  in 
order  to  get  the  best  understanding  of  the  applicant’s  competencies,  agencies  should  use  a 
multi-method  approach  in  evaluating  applicants.  Additionally,  when  deciding  on  the 
assessment  tools  to  use,  agencies  should  select  assessment  tools  that  complement  each 
other  and  not  just  provide  the  same  information  in  a  different  or  disjointed  manner.  This 
multiple  hurdle  and  multiple  assessment  tool  approach  allows  for  the  evaluation  of 
applicants  by  multiple  evaluators  and  methods.  Lastly,  this  approach  provides  more 

process  validity,  which  is  important  for  morale  of  current  and  future  applicants,  as  well  as 
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the  employees  who  will  report  to  these  new  supervisors.  If  an  agency  hopes  to  recruit  the 
best  employees  for  first-line  supervisor  positions,  the  process  must  be  fair,  valid, 
transparent  and  select  the  best  qualified  candidates. 

F.  MEASURING  FIRST  LINE  SUPERVISORS 

H.  James  Harrington  stated: 

Measurement  is  the  first  step  that  leads  to  control  and  eventually  to 
improvement.  If  you  can’t  measure  something,  you  can’t  understand  it.  If 
you  can’t  understand  it,  you  can’t  control  it.  If  you  can’t  control  it,  you 
can’t  improve  it. 

Perhaps  the  government’s  inability  to  measure  first-line  supervisor  performance  is 
the  reason  why  the  effectiveness  of  first-line  supervisors  in  the  federal  government 
continues  to  be  called  into  question.  Although  the  best  practices  mentioned  above  will 
potentially  assist  in  selecting  effective  first-line  supervisors,  in  order  to  allow  agencies  to 
determine  if  changes  to  first-line  supervisor  selection  processes  are  working,  there  must 
be  a  method  to  gauge  and  monitor  the  effectiveness  of  first-line  supervisors. 

This  study  used  the  perceptions  of  nonsupervisory  employees,  measured  by  the 
First-Line  Supervisor  Index  (FLSI),  to  determine  the  effectiveness  of  first-line 
supervisors  in  the  agencies  examined.  Although  the  validity  of  perceptive  measures  can 
be  questioned,  these  tools  have  been  shown  to  have  a  high  positive  correlation  to  other 
measures  of  performance  that  can  be  used  in  examining  the  effectiveness  of  first-line 
supervisors.  As  described  in  Chapter  II,  a  multi-method  approach  is  perhaps  the  most 
valid  method  to  measure  perfonnance  and  effectiveness;  however,  when  examining  first- 
line  supervisors  in  the  federal  government,  there  is  currently  no  other  data  available  that 
can  be  used  that  examines  effectiveness  of  first-line  supervisors  at  the  micro  level. 
Therefore,  potential  questions  about  the  usefulness  of  perceptive  measures  should  not 
stop  the  federal  government  from  using  these  perceptions  to  measure  perfonnance  and 
effectiveness  of  first-line  supervisors. 

The  FLSI  used  in  this  study  was  a  combination  of  eight  questions  (see  Table  7 

and  Appendix  A)  that  were  consistently  used  in  the  Federal  Human  Capital  Survey 

(FHCS)  in  2006  and  2008,  in  the  Federal  Employee  Viewpoint  Survey  (FedView)  in 
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2010  and  focused  on  specific  behaviors  and  outcomes  of  first-line  supervisors.  In  2010, 
as  part  of  the  FedView,  OPM  created  a  “My  Supervisor/Team  Leader”  sub-index  that 
contained  the  eight  questions  consistent  throughout  the  three  examined  in  this  study  and 
three  additional  questions. 


Table  7.  Supervisor/Team  Leader  Questions 


My  Supervisor/Team  Leader  Sub-Index 

2006/2008 

2010 

My  supervisor  supports  my  need  to  balance  work  and  other  life  issues. 

V 

V 

My  supervisor/team  leader  provides  me  with  opportunities  to  demonstrate  my 
leadership  skills 

V 

V 

Discussions  with  my  supervisor/team  leader  about  my  performance  are 
worthwhile. 

V 

V 

My  supervisor/team  leader  is  committed  to  work  force  representative  of  all 
segments  of  society. 

V 

V 

My  supervisor/team  leader  provides  me  with  constructive  suggestions  to 
improve  my  job  performance. 

V 

V 

Supervisors/team  leaders  in  my  work  unit  support  employee  development. 

V 

V 

My  supervisor/team  leader  listens  to  what  I  have  to  say. 

V 

My  supervisor/team  leader  treats  me  with  respect. 

V 

In  the  last  six  months,  my  supervisor/team  leader  has  talked  with  me  about 
my  performance. 

V 

1  have  trust  and  confidence  in  my  supervisor. 

V 

V 

Overall,  how  good  a  job  do  you  feel  is  being  done  by  your  immediate 
supervisor/team  leader? 

V 

V 

By  using  the  11  questions  from  “My  Supervisor/Team  Leader”  sub-index, 
agencies  can  begin  to  track  and  measure  first-line  supervisor  performance  and 
effectiveness.  Since  OPM  has  begun  administering  the  FedView  every  year  starting  in 
2010,  agencies  can  track  first-line  supervisor  performance  and  effectiveness  as  part  of  a 
yearly  review.  This  will  provide  immediate  feedback  and  insight  to  any  changes  that 
agencies  might  have  made  in  the  first-line  supervisor  selection  processes  or  other  policy 
changes  that  impact  first-line  supervisor  performance  and  effectiveness  within  their 
agency. 


G.  THE  COST  OF  SELECTING  EFFECTIVE  FIRST-LINE  SUPERVISORS 

During  the  interviews  conducted  for  this  study,  human  resource  specialists 
expressed  concern  about  the  cost  associated  with  selection  processes.  Many  persons 
interviewed  reported  cost  often  impacted  the  choice  of  assessment  methods  used  in  first- 
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line  supervisor  selection.  Recent  budget  cuts  will  continue  to  make  the  cost  associated 
with  selection  processes  an  issue  of  concern  for  agencies.  Therefore,  DHS,  which  had 
the  lower  FLSI  scores,  must  look  for  innovative  ways  to  allow  agencies  to  use  needed 
assessment  tools  but  keep  costs  down.  Although  there  are  numerous  innovative  ways  to 
cut  costs  for  selection  processes,  this  study  revealed  two  that  are  of  note. 

First,  by  implementing  a  standard  first  hurdle  psychometric  assessment  test,  the 
costs  potentially  can  be  reduced  for  agencies  within  DHS  depending  on  how  they  plan  to 
use  the  test  in  the  overall  selection  process.  Second,  DHS  human  resource 
representatives  should  coordinate  with  each  other  to  share  assessment  methods  they  have 
in  common.  Although  this  study  revealed  that  CBP  and  ICE  have  less  effective  first-line 
supervisors  compared  to  the  other  agencies  examined,  a  best  practice  in  potentially 
saving  money  was  revealed.  CBP  and  ICE  reduce  the  cost  of  first-line  supervisor 
selection  by  sharing  common  selection  methods  for  first-line  supervisors.  This  practice 
could  be  emulated  by  all  DHS  agencies  as  a  strategy  to  cut  down  costs  associated  with 
implementing  best  practices  for  first-line  supervisor  selection.  Additionally,  by  sharing 
costs,  resources  and  assessment  tools,  the  use  of  common  best  practice  standards  and 
connectedness  among  agencies  within  DHS  will  be  enhanced. 

Clearly,  the  agencies  within  DHS  need  to  find  ways  to  implement  best  practices  to 
improve  selection  processes  for  first-line  supervisors  while  keeping  the  financial  burden 
for  any  one  agency  at  a  minimum.  Senior  executives  of  each  DHS  agency  need  to 
remember  the  potential  cost  of  poor  or  ineffective  first-line  supervisor  selection 
processes  goes  beyond  a  human  resource  budget  allocation;  that  is;  the  loss  of  employees 
supervised  by  poor  first-line  supervisors.  Ineffective  selection  of  first-line  supervisors, 
thus,  can  have  a  cascading  effect  that  is  hard  to  quantify,  but  potentially  very  costly 
because  first-line  supervisors  are  crucial  to  the  day-to-day  operation  of  any  organization. 
They  are  directly  responsible  for  day-to-day  leadership,  coaching,  mentoring,  job 
performance  and  retention  of  nonsupervisory  employees  (Human  Resources  Management 
Panel,  2003),  and  they  ensure  that  the  vision,  guidance  and  direction  of  executive  leaders 
become  reality  in  an  organization.  It  is  reasonable  to  say  that  the  cost  paid  by  an  agency 
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that  selects  ineffective  supervisors,  grossly  outweighs  the  costs  associated  with 
implementing  an  effective  first-line  supervisor  selection  process. 


H.  FUTURE  RESEARCH 

Several  important  areas  of  further  research  should  be  pursued.  First,  researchers 
should  try  to  improve  the  perceptive  measures  of  first-line  supervisor  effectiveness 
(FLSI)  used  in  this  study  or  create  a  new,  more  effective  measure  that  would  allow  first- 
line  supervisor  performance  to  be  assessed  on  a  regular  basis. 

Correspondence  between  perceptive  measures  of  effectiveness  and  more  objective 
measures  of  effectiveness  needs  to  be  explored  as  well.  Mapping  out  the  links  between 
specific  behaviors  and  outcomes  like  effectiveness  would  be  very  helpful  for  designing 
first-line  supervisor  training  activities. 

Researchers  should  also  examine  what  effect  substantial  changes  in  first-line 
supervisor  selection  process  might  have  on  how  an  agency’s  supervisors  are  rated  on  the 
FLSI. 

Lastly,  first-line  supervisor  selection  best  practices  should  be  identified  for  any  of 
these  follow-on  research  efforts.  Application  of  these  best  practices  in  specific  agencies 
should  be  favorably  considered  to  improve  the  overall  quality  of  supervision  in  the 
federal  government. 
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APPENDIX  A.  FIRST-LINE  SUPERVISOR  INDEX  SURVEY 

INFORMATION 


Table  8.  First-Line  Supervisor  Index  Questions 


First-Line  Supervisor  Index  Questions 

Survey  Question 
Number 

2006 

2008 

2010 

12 

12 

42 

My  supervisor  supports  my  need  to  balance  work  and  other  life 
issues. 

13 

13 

43 

My  supervisor/team  leader  provides  me  with  opportunities  to 
demonstrate  my  leadership  skills. 

31 

31 

44 

Discussions  with  my  supervisor/team  leader  about  my  performance 
are  worthwhile. 

33 

34 

45 

My  supervisor/team  leader  is  committed  to  a  workforce 
representative  of  all  segments  of  society. 

47 

48 

46 

My  supervisor/team  leader  provides  me  with  constructive 
suggestions  to  improve  my  job  performance. 

48 

49 

47 

Supervisors/team  leaders  in  my  work  unit  support  employee 
development. 

7 

7 

51 

I  have  trust  and  confidence  in  my  supervisor. 

9 

9 

52 

Overall,  how  good  a  job  do  you  feel  is  being  done  by  your 
immediate  supervisor/team  leader? 

Table  9.  First-Line  Supervisor  Demographic  Characteristics 


Demographic  Characteristics 

What  is  your  supervisory  status?  (DSUPER) 

2010 

Nonsupervisor/Team  Leader 

Supervisor 

Manager/Executive 

2006 

& 

2008 

Nonsupervisor 

Team  Leader 

Supervisor 

Manager 

Executive  Note:  All  2006  and  2008  data  was  re-coded  to  match  the  2010 

DSUPER  selection  options 
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Table  10.  First-Line  Supervisor  Index  Value  Labels 


Value  Labels  For  FLSI  Items 

Question  Num 

ter  Per  Year 

2006 

2008 

2010 

12 

12 

42 

5=Strongly  Agree 

4=Agree 

3— Neither  Agree  nor  Disagree 

2=Disagree 
l=Strongly  Disagree 

X=Do  Not  Know 

13 

13 

43 

31 

31 

44 

33 

34 

45 

47 

48 

46 

48 

49 

47 

7 

7 

51 

5=Strongly  Agree 

4=Agree 

3=Neither  Agree  nor  Disagree 

2=Disagree 

1  ^Strongly  Disagree 

9 

9 

52 

5=Very  Good 

4=Good 

3=Fair 

2=Poor 

l=Very  Poor 

Table  1 1 .  Agency  Survey  Code  for  Federal  Human  Capital  Survey  and  Federal 

Viewpoint  Survey 


Survey  Code 

Agency 

DEA 

Drug  Enforcement  Administration  (DEA) 

DJ02 

Federal  Bureau  of  Investigation  (FBI) 

DJ08 

U.S.  Marshals  Service  (USMS) 

DJI  5 

Alcohol,  Tobacco,  Firearms,  and  Explosives  (ATF) 

HSAB 

Citizenship  and  Immigration  Services  (CIS) 

HSAC 

U.S.  Coast  Guard  (CG) 

HSAD 

U.S.  Secret  Service  (SS) 

HSBB 

Immigration  and  Customs  Enforcement  (ICE) 

HSBC 

Transportation  Security  Administration  (TSA) 

HSBD 

Customs  and  Border  Protection  (CBP) 

HSCB 

Federal  Emergency  Management  Agency  (FEMA) 

58 


APPENDIX  B.  FIRST-LINE  SUPERVISOR  SELECTION  PROCESS  BENCHMARKS 


Table  12.  Benchmarks  for  First-Line  Supervisor  Selection  Processes 


First-Line  Supervisor  Selection  Process 


Hurdles 

ATF 

DEA 

FBI 

ICE 

CBP 

TSA 

N  on-La w  Enfor 

Special  Agent 

N  o n - La w  Enfor 

Special  Agent 

Special  Agent 

N  o  n  -  La  w  Enfor 

Special  Agent 

0  fficer/A  gricul 

Border  Patrol 

N  on-Law  Enfor 

Air  M  arshal 

1st  Hurdle 

KSA  and  Resume 
Application  Package 

Assessment  Center- 

(Oral  Presentation, 
Simulations,  Panel 
Interview) 

KSA  and  Resume 
Application  Package 

Supervisor  Rating 

Leadership  Skills 
Assessment  In-Baske 

Simulation 

KSA  and  Resume 
Application  Package 

Logic  Reasoning  Test 

Logic  Reasoning  Test 

Logic  Reasoning  Test 

KSA  and  Resume 
Application  Package 

KSA  and  Resume 
Application  Package 

Assessment  Center- 
(Panel  interview, 
Simulations,  Written 
Test) 

Multiple  Choice 
Writing  Test 

Multiple  Choice 
Writing  Test 

Multiple  Choice 
Writing  Test 

Supvervisor  Rating 

Computer-Based  In- 
Basket  Simulation 

In-Basket  Simulation 

In-Basket  Simulation 

Job-knowledge  Test 

Career  Experience 
Invetory 

Career  Experience 
Invetory 

Job-Knowledge  Test 

Administration 

Electronic 

Submission 

In-person/Computer 

Based 

Electronic 

Submission 

In-person/Computer 

Based 

3  Person  Panel- 
Telephone 

Electronic 

Submission 

Electronic 

Submission 

Electronic 

Submission 

Electronic 

Submission 

Electronic 

Submission 

Electronic 

Submission 

2nd  Hurdle 

Struture  Interview 

Struture  Interview 

HQ  Career  Board 

Competencies 

Narrative  Reviewed 
by  SES-Level  Board 

Struture  Interview 

Struture  Interview 

HQ  Career  Board 

Administration 

In-Person  at  Field 

Office 

In-Person  at  Field 

Office 

HQ  Panel 

HQ  Career  Board 

In-Person  at  Field 

Office 

In-Person  at  Field 

Office 

HQ  3  Person  Panel 
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APPENDIX  C.  ANOVA  TABLES/GRAPHS 


Table  13.  ANOVA  2006 


ANOVA  2006 

Sum  of 
Squares 

df 

Mean  Square 

F 

Sig. 

FLSI  Between  Groups 

18534.990 

10 

1853.499 

32.925 

.000 

Within  Groups 

388993.442 

6910 

56.294 

Total 

407528.432 

6920 

Table  14.  ANOVA  2008 


ANOVA  2008 

Sum  of 
Squares 

df 

Mean  Square  F  Sig. 

FLSI 

Between  Groups 

13915.978 

10 

1391.598  23.209  .000 

Within  Groups 

474997.396 

7922 

59.959 

Total 

488913.374 

7932 

Table  15.  ANOVA2010 


ANOVA  2010 

Sum  of 
Squares 

df 

Mean  Square 

F 

Sig. 

FLSI  Between  Groups 

10393.630 

10 

1039.363 

14.604 

.000 

Within  Groups 

639606.256 

8987 

71.170 

Total 

649999.886 

8997 
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APPENDIX  D 


FLSI  CORRELATION  TABLES 


Table  16.  2006  Correlation  Tables 


DEA  2006 

Behavior 

Factor 

Outcome 

Factor 

FLSI 

Behavior  Factor 

Pearson  Correlation 

1 

,787(**) 

,982(**) 

Sig.  (2-tailed) 

.000 

.000 

N 

1175 

1175 

1175 

Outcome  Factor 

Pearson  Correlation 

,787(**) 

1 

,890(**) 

Sig.  (2-tailed) 

.000 

.000 

N 

1175 

1175 

1175 

FLSI18 

Pearson  Correlation 

.982(**) 

,890(**) 

1 

Sig.  (2-tailed) 

.000 

.000 

N 

1175 

1175 

1175 

Behavior 

Outcome 

FBI  2006 

Factor 

Factor 

FLSI 

Behavior  Factor 

Pearson  Correlation 

1 

,725(**) 

,973(**) 

Sig.  (2-tailed) 

.000 

.000 

N 

422 

420 

420 

Outcome  Factor 

Pearson  Correlation 

,725(**) 

1 

,864(**) 

Sig.  (2-tailed) 

.000 

.000 

N 

420 

432 

420 

FLSI 

Pearson  Correlation 

,973(**) 

,864(**) 

1 

Sig.  (2-tailed) 

.000 

.000 

N 

420 

420 

420 

Behavior 

Outcome 

USMS  2006 

Factor 

Factor 

FLSI 

Behavior  Factor 

Pearson  Correlation 

1 

,762(**) 

,980(**) 

Sig.  (2-tailed) 

.000 

.000 

N 

530 

530 

530 

Outcome  Factor 

Pearson  Correlation 

,762(**) 

1 

.876(**) 

Sig.  (2-tailed) 

.000 

.000 

N 

530 

530 

530 

FLSI 

Pearson  Correlation 

,980(**) 

.876(**) 

1 

Sig.  (2-tailed) 

.000 

.000 

N 

530 

530 

530 

**  Correlation  is  significant  at  the  0.01  level  (2-tailed). 


The  FLSI  is  the  aggregate  mean  of  eight  questions  concerning  first-line  supervisor  behavior  s  and 
outcomes.  The  behavior  factor  is  the  aggregate  mean  of  behaviors  (six  questions)  and  the  outcome  factor  is 
the  aggregate  mean  of  the  outcomes  (two  questions)  in  the  FLSI. 


65 


ATF  2006 

Behavior 

Factor 

Outcome 

Factor 

FLSI 

Behavior  Factor 

Pearson  Correlation 

1 

,803(**) 

,983(**) 

Sig.  (2-tailed) 

.000 

.000 

N 

557 

557 

557 

Outcome  Factor 

Pearson  Correlation 

,803(**) 

1 

,898(**) 

Sig.  (2-tailed) 

.000 

.000 

N 

557 

557 

557 

FLSI 

Pearson  Correlation 

,983(**) 

,898(**) 

1 

Sig.  (2-tailed) 

.000 

.000 

N 

557 

557 

557 

Behavior 

Outcome 

CIS  2006 

Factor 

Factor 

FLSI 

Behavior  Factor 

Pearson  Correlation 

1 

,752(**) 

,981(**) 

Sig.  (2-tailed) 

.000 

.000 

N 

647 

647 

647 

Outcome  Factor 

Pearson  Correlation 

,752(**) 

1 

,866(**) 

Sig.  (2-tailed) 

.000 

.000 

N 

647 

647 

647 

FLSI 

Pearson  Correlation 

,981(**) 

,866(**) 

1 

Sig.  (2-tailed) 

.000 

.000 

N 

647 

647 

647 

Behavior 

Outcome 

USCG  2006 

Factor 

Factor 

FLSI 

Behavior  Factor 

Pearson  Correlation 

1 

,804(**) 

.985(**) 

Sig.  (2-tailed) 

.000 

.000 

N 

695 

695 

695 

Outcome  Factor 

Pearson  Correlation 

,804(**) 

1 

,895(**) 

Sig.  (2-tailed) 

.000 

.000 

N 

695 

695 

695 

FLSI 

Pearson  Correlation 

,985(**) 

,895(**) 

1 

Sig.  (2-tailed) 

.000 

.000 

N 

695 

695 

695 

**  Correlation  is  significant  at  the  0.01  level  (2-tailed). 
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USSS  2006 

Behavior 

Factor 

Outcome 

Factor 

FLSI 

Behavior  Factor 

Pearson  Correlation 

1 

,764(**) 

,980(**) 

Sig.  (2-tailed) 

.000 

.000 

N 

537 

537 

537 

Outcome  Factor 

Pearson  Correlation 

,764(**) 

1 

,878(**) 

Sig.  (2-tailed) 

.000 

.000 

N 

537 

537 

537 

FLSI 

Pearson  Correlation 

,980(**) 

,878(**) 

1 

Sig.  (2-tailed) 

.000 

.000 

N 

537 

537 

537 

Behavior 

Outcome 

ICE  2006 

Factor 

Factor 

FLSI 

Behavior  Factor 

Pearson  Correlation 

1 

.758(**) 

,980(**) 

Sig.  (2-tailed) 

.000 

.000 

N 

604 

604 

604 

Outcome  Factor 

Pearson  Correlation 

,758(**) 

1 

,872(**) 

Sig.  (2-tailed) 

.000 

.000 

N 

604 

604 

604 

FLSI 

Pearson  Correlation 

,980(**) 

,872(**) 

1 

Sig.  (2-tailed) 

.000 

.000 

N 

604 

604 

604 

Behavior 

Outcome 

TSA  2006 

Factor 

Factor 

FLSI 

Behavior  Factor 

Pearson  Correlation 

1 

,804(**) 

,984(**) 

Sig.  (2-tailed) 

.000 

.000 

N 

453 

448 

448 

Outcome  Factor 

Pearson  Correlation 

,804(**) 

1 

,896(**) 

Sig.  (2-tailed) 

.000 

.000 

N 

448 

458 

448 

FLSI 

Pearson  Correlation 

,984(**) 

,896(**) 

1 

Sig.  (2-tailed) 

.000 

.000 

N 

448 

448 

448 

**  Correlation  is  significant  at  the  0.01  level  (2-tailed). 
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CBP  2006 

Behavior 

Factor 

Outcome 

Factor 

FLSI 

Behavior  Factor 

Pearson  Correlation 

1 

,752(**) 

,980(**) 

Sig.  (2-tailed) 

.000 

.000 

N 

847 

847 

847 

Outcome  Factor 

Pearson  Correlation 

,752(**) 

1 

,867(**) 

Sig.  (2-tailed) 

.000 

.000 

N 

847 

847 

847 

FLSI 

Pearson  Correlation 

,980(**) 

,867(**) 

1 

Sig.  (2-tailed) 

.000 

.000 

N 

847 

847 

847 

Behavior 

Outcome 

FEMA  2006 

Factor 

Factor 

FLSI 

Behavior  Factor 

Pearson  Correlation 

1 

.81 1(**) 

,983(**) 

Sig.  (2-tailed) 

.000 

.000 

N 

461 

461 

461 

Outcome  Factor 

Pearson  Correlation 

.81 1(**) 

1 

,904(**) 

Sig.  (2-tailed) 

.000 

.000 

N 

461 

461 

461 

FLSI 

Pearson  Correlation 

,983(**) 

,904(**) 

1 

Sig.  (2-tailed) 

.000 

.000 

N 

461 

461 

461 

**  Correlation  is  significant  at  the  0.01  level  (2-tailed). 


Table  17.  2008  Correlation  Tables 


DEA  2008 

Behavior 

Factor 

Outcome 

Factor 

FLSI 

Behavior  Factor 

Pearson  Correlation 

1 

,815(**) 

,984(**) 

Sig.  (2-tailed) 

.000 

.000 

N 

936 

936 

936 

Outcome  Factor 

Pearson  Correlation 

,815(**) 

1 

.905(**) 

Sig.  (2-tailed) 

.000 

.000 

N 

936 

936 

936 

FLSI 

Pearson  Correlation 

,984(**) 

.905(**) 

1 

Sig.  (2-tailed) 

.000 

.000 

N 

936 

936 

936 
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FBI  2008 

Behavior 

Factor 

Outcome 

Factor 

FLSI 

Behavior  Factor 

Pearson  Correlation 

1 

,701(**) 

,971(**) 

Sig.  (2-tailed) 

.000 

.000 

N 

305 

302 

302 

Outcome  Factor 

Pearson  Correlation 

,701(**) 

1 

,851(**) 

Sig.  (2-tailed) 

.000 

.000 

N 

302 

309 

302 

FLSI 

Pearson  Correlation 

,971(**) 

,851(**) 

1 

Sig.  (2-tailed) 

.000 

.000 

N 

302 

302 

302 

Behavior 

Outcome 

USMS  2008 

Factor 

Factor 

FLSI 

Behavior  Factor 

Pearson  Correlation 

1 

,823(**) 

.985(**) 

Sig.  (2-tailed) 

.000 

.000 

N 

1471 

1471 

1471 

Outcome  Factor 

Pearson  Correlation 

,823(**) 

1 

,909(**) 

Sig.  (2-tailed) 

.000 

.000 

N 

1471 

1471 

1471 

FLSI 

Pearson  Correlation 

,985(**) 

,909(**) 

1 

Sig.  (2-tailed) 

.000 

.000 

N 

1471 

1471 

1471 

Behavior 

Outcome 

ATF  2008 

Factor 

Factor 

FLSI 

Behavior  Factor 

Pearson  Correlation 

1 

,842(**) 

,988(**) 

Sig.  (2-tailed) 

.000 

.000 

N 

405 

405 

405 

Outcome  Factor 

Pearson  Correlation 

,842(**) 

1 

,916(**) 

Sig.  (2-tailed) 

.000 

.000 

N 

405 

405 

405 

FLSI 

Pearson  Correlation 

,988(**) 

,916(**) 

1 

Sig.  (2-tailed) 

.000 

.000 

N 

405 

405 

405 

**  Correlation  is  significant  at  the  0.01  level  (2-tailed). 
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CIS  2008 

Behavior 

Factor 

Outcome 

Factor 

FLSI 

Behavior  Factor 

Pearson  Correlation 

1 

.777(**) 

,983(**) 

Sig.  (2-tailed) 

.000 

.000 

N 

417 

417 

417 

Outcome  Factor 

Pearson  Correlation 

.777(**) 

1 

,880(**) 

Sig.  (2-tailed) 

.000 

.000 

N 

417 

417 

417 

FLSI 

Pearson  Correlation 

,983(**) 

,880(**) 

1 

Sig.  (2-tailed) 

.000 

.000 

N 

417 

417 

417 

Behavior 

Outcome 

USCG  2008 

Factor 

Factor 

FLSI 

Behavior  Factor 

Pearson  Correlation 

1 

.796(**) 

,984(**) 

Sig.  (2-tailed) 

.000 

.000 

N 

539 

539 

539 

Outcome  Factor 

Pearson  Correlation 

,796(**) 

1 

,892(**) 

Sig.  (2-tailed) 

.000 

.000 

N 

539 

539 

539 

FLSI 

Pearson  Correlation 

,984(**) 

,892(**) 

1 

Sig.  (2-tailed) 

.000 

.000 

N 

539 

539 

539 

Behavior 

Outcome 

USSS  2008 

Factor 

Factor 

FLSI 

Behavior  Factor 

Pearson  Correlation 

1 

,774(**) 

,983(**) 

Sig.  (2-tailed) 

.000 

.000 

N 

359 

359 

359 

Outcome  Factor 

Pearson  Correlation 

.774(**) 

1 

,878(**) 

Sig.  (2-tailed) 

.000 

.000 

N 

359 

359 

359 

FLSI 

Pearson  Correlation 

,983(**) 

,878(**) 

1 

Sig.  (2-tailed) 

.000 

.000 

N 

359 

359 

359 

**  Correlation  is  significant  at  the  0.01  level  (2-tailed). 
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ICE  2008 

Behavior 

Factor 

Outcome 

Factor 

FLSI 

Behavior  Factor 

Pearson  Correlation 

1 

,781(**) 

,981(**) 

Sig.  (2-tailed) 

.000 

.000 

N 

370 

370 

370 

Outcome  Factor 

Pearson  Correlation 

,781(**) 

1 

,886(**) 

Sig.  (2-tailed) 

.000 

.000 

N 

370 

370 

370 

FLSI 

Pearson  Correlation 

,981(**) 

,886(**) 

1 

Sig.  (2-tailed) 

.000 

.000 

N 

370 

370 

370 

Behavior 

Outcome 

TSA  2008 

Factor 

Factor 

FLSI 

Behavior  Factor 

Pearson  Correlation 

1 

,794(**) 

,984(**) 

Sig.  (2-tailed) 

.000 

.000 

N 

1169 

1169 

1169 

Outcome  Factor 

Pearson  Correlation 

,794(**) 

1 

,889(**) 

Sig.  (2-tailed) 

.000 

.000 

N 

1169 

1169 

1169 

FLSI 

Pearson  Correlation 

,984(**) 

,889(**) 

1 

Sig.  (2-tailed) 

.000 

.000 

N 

1169 

1169 

1169 

Behavior 

Outcome 

CBP  2008 

Factor 

Factor 

FLSI 

Behavior  Factor 

Pearson  Correlation 

1 

,787(**) 

,983(**) 

Sig.  (2-tailed) 

.000 

.000 

N 

1606 

1606 

1606 

Outcome  Factor 

Pearson  Correlation 

,787(**) 

1 

,886(**) 

Sig.  (2-tailed) 

.000 

.000 

N 

1606 

1606 

1606 

FLSI 

Pearson  Correlation 

,983(**) 

,886(**) 

1 

Sig.  (2-tailed) 

.000 

.000 

N 

1606 

1606 

1606 

**  Correlation  is  significant  at  the  0.01  level  (2-tailed). 
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FEMA  2008 

Behavior 

Factor 

Outcome 

Factor 

FLSI 

Behavior  Factor 

Pearson  Correlation 

1 

797(**) 

,982(**) 

Sig.  (2-tailed) 

.000 

.000 

N 

359 

359 

359 

Outcome  Factor 

Pearson  Correlation 

797(**) 

1 

,897(**) 

Sig.  (2-tailed) 

.000 

.000 

N 

359 

359 

359 

FLSI 

Pearson  Correlation 

,982(**) 

,897(**) 

1 

Sig.  (2-tailed) 

.000 

.000 

N 

359 

359 

359 

**  Correlation  is  significant  at  the  0.01  level  (2-tailed). 


Table  18.  2010  Correlation  Tables 


DEA  2010 

Behavior 

Factor 

Outcome 

Factor 

FLSI 

Behavior  Factor 

Pearson  Correlation 

1 

,843(**) 

,988(**) 

Sig.  (2-tailed) 

.000 

.000 

N 

1171 

1162 

1162 

Outcome  Factor 

Pearson  Correlation 

,843(**) 

1 

,914(**) 

Sig.  (2-tailed) 

.000 

.000 

N 

1162 

1181 

1162 

FLSI 

Pearson  Correlation 

,988(**) 

,914(**) 

1 

Sig.  (2-tailed) 

.000 

.000 

N 

1162 

1162 

1162 

Behavior 

Outcome 

FBI  2010 

Factor 

Factor 

FLSI 

Behavior  Factor 

Pearson  Correlation 

1 

,781(**) 

.985(**) 

Sig.  (2-tailed) 

.000 

.000 

N 

240 

234 

234 

Outcome  Factor 

Pearson  Correlation 

,781(**) 

1 

,877(**) 

Sig.  (2-tailed) 

.000 

.000 

N 

234 

240 

234 

FLSI 

Pearson  Correlation 

,985(**) 

,877(**) 

1 

Sig.  (2-tailed) 

.000 

.000 

N 

234 

234 

234 

**  Correlation  is  significant  at  the  0.01  level  (2-tailed). 
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USMS  2010 

Behavior 

Factor 

Outcome 

Factor 

FLSI 

Behavior  Factor 

Pearson  Correlation 

1 

,849(**) 

,989(**) 

Sig.  (2-tailed) 

.000 

.000 

N 

1641 

1628 

1628 

Outcome  Factor 

Pearson  Correlation 

,849(**) 

1 

,918(**) 

Sig.  (2-tailed) 

.000 

.000 

N 

1628 

1647 

1628 

FLSI 

Pearson  Correlation 

,989(**) 

,918(**) 

1 

Sig.  (2-tailed) 

.000 

.000 

N 

1628 

1628 

1628 

Behavior 

Outcome 

ATF  2010 

Factor 

Factor 

FLSI 

Behavior  Factor 

Pearson  Correlation 

1 

,849(**) 

,989(**) 

Sig.  (2-tailed) 

.000 

.000 

N 

1653 

1644 

1644 

Outcome  Factor 

Pearson  Correlation 

,849(**) 

1 

,917(**) 

Sig.  (2-tailed) 

.000 

.000 

N 

1644 

1664 

1644 

FLSI 

Pearson  Correlation 

,989(**) 

,917(**) 

1 

Sig.  (2-tailed) 

.000 

.000 

N 

1644 

1644 

1644 

Behavior 

Outcome 

CIS  2010 

Factor 

Factor 

FLSI 

Behavior  Factor 

Pearson  Correlation 

1 

.836(**) 

,989(**) 

Sig.  (2-tailed) 

.000 

.000 

N 

662 

658 

658 

Outcome  Factor 

Pearson  Correlation 

,836(**) 

1 

,908(**) 

Sig.  (2-tailed) 

.000 

.000 

N 

658 

663 

658 

FLSI 

Pearson  Correlation 

,989(**) 

,908(**) 

1 

Sig.  (2-tailed) 

.000 

.000 

N 

658 

658 

658 

**  Correlation  is  significant  at  the  0.01  level  (2-tailed). 
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USCG  2010 

Behavior 

Factor 

Outcome 

Factor 

FLSI 

Behavior  Factor 

Pearson  Correlation 

1 

,806(**) 

,986(**) 

Sig.  (2-tailed) 

.000 

.000 

N 

554 

549 

549 

Outcome  Factor 

Pearson  Correlation 

,806(**) 

1 

,893(**) 

Sig.  (2-tailed) 

.000 

.000 

N 

549 

558 

549 

FLSI 

Pearson  Correlation 

.986(**) 

,893(**) 

1 

Sig.  (2-tailed) 

.000 

.000 

N 

549 

549 

549 

Behavior 

Outcome 

USSS  2010 

Factor 

Factor 

FLSI 

Behavior  Factor 

Pearson  Correlation 

1 

,810(**) 

,985(**) 

Sig.  (2-tailed) 

.000 

.000 

N 

292 

287 

287 

Outcome  Factor 

Pearson  Correlation 

,810(**) 

1 

,898(**) 

Sig.  (2-tailed) 

.000 

.000 

N 

287 

291 

287 

FLSI 

Pearson  Correlation 

,985(**) 

,898(**) 

1 

Sig.  (2-tailed) 

.000 

.000 

N 

287 

287 

287 

Behavior 

Outcome 

ICE  2010 

Factor 

Factor 

FLSI 

Behavior  Factor 

Pearson  Correlation 

1 

,809(**) 

,987(**) 

Sig.  (2-tailed) 

.000 

.000 

N 

414 

414 

414 

Outcome  Factor 

Pearson  Correlation 

,809(**) 

1 

,893(**) 

Sig.  (2-tailed) 

.000 

.000 

N 

414 

418 

414 

FLSI 

Pearson  Correlation 

,987(**) 

,893(**) 

1 

Sig.  (2-tailed) 

.000 

.000 

N 

414 

414 

414 

**  Correlation  is  significant  at  the  0.01  level  (2-tailed). 
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TSA  2010 

Behavior 

Factor 

Outcome 

Factor 

FLSI 

Behavior  Factor 

Pearson  Correlation 

1 

,827(**) 

,988(**) 

Sig.  (2-tailed) 

.000 

.000 

N 

1167 

1156 

1156 

Outcome  Factor 

Pearson  Correlation 

,827(**) 

1 

,903(**) 

Sig.  (2-tailed) 

.000 

.000 

N 

1156 

1179 

1156 

FLSI 

Pearson  Correlation 

,988(**) 

,903(**) 

1 

Sig.  (2-tailed) 

.000 

.000 

N 

1156 

1156 

1156 

Behavior 

Outcome 

CBP  2010 

Factor 

Factor 

FLSI 

Behavior  Factor 

Pearson  Correlation 

1 

,820(**) 

,989(**) 

Sig.  (2-tailed) 

.000 

.000 

N 

953 

943 

943 

Outcome  Factor 

Pearson  Correlation 

,820(**) 

1 

,896(**) 

Sig.  (2-tailed) 

.000 

.000 

N 

943 

970 

943 

FLSI 

Pearson  Correlation 

,989(**) 

,896(**) 

1 

Sig.  (2-tailed) 

.000 

.000 

N 

943 

943 

943 

Behavior 

Outcome 

FEMA  2010 

Factor 

Factor 

FLSI 

Behavior  Factor 

Pearson  Correlation 

1 

,854(**) 

,989(**) 

Sig.  (2-tailed) 

.000 

.000 

N 

328 

323 

323 

Outcome  Factor 

Pearson  Correlation 

,854(**) 

1 

,921(**) 

Sig.  (2-tailed) 

.000 

.000 

N 

323 

329 

323 

FLSI 

Pearson  Correlation 

,989(**) 

,921(**) 

1 

Sig.  (2-tailed) 

.000 

.000 

N 

323 

323 

323 

**  Correlation  is  significant  at  the  0.01  level  (2-tailed). 
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